Kubernetes | Dosu

This guide provides comprehensive instructions for deploying DaemonEye on Kubernetes, including manifests, Helm charts, and production deployment strategies.

Kubernetes Overview#

DaemonEye is designed to run efficiently on Kubernetes, providing:

Scalability: Horizontal pod autoscaling and cluster-wide deployment
High Availability: Multi-replica deployments with health checks
Security: RBAC, network policies, and pod security standards
Observability: Prometheus metrics, structured logging, and distributed tracing
Management: Helm charts and GitOps integration

Architecture Components#

procmond: DaemonSet for process monitoring on each node
daemoneye-agent: Deployment for alerting and orchestration
daemoneye-cli: Job/CronJob for management tasks
Security Center: Deployment for web-based management (Business/Enterprise)

Prerequisites#

Cluster Requirements#

Minimum Requirements:

Kubernetes 1.20+
2+ worker nodes
4+ CPU cores total
8+ GB RAM total
50+ GB storage
Recommended Requirements:
Kubernetes 1.24+
3+ worker nodes
8+ CPU cores total
16+ GB RAM total
100+ GB storage

Required Tools#

# Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Install Helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Install kustomize
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash

Basic Deployment#

Namespace and RBAC#

namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: daemoneye
  labels:
    name: daemoneye
    app.kubernetes.io/name: daemoneye
    app.kubernetes.io/version: 1.0.0

rbac.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: daemoneye-procmond
  namespace: daemoneye
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: daemoneye-agent
  namespace: daemoneye
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: daemoneye-procmond
rules:
  - apiGroups: [""]
    resources: ["nodes", "pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: daemoneye-procmond
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: daemoneye-procmond
subjects:
  - kind: ServiceAccount
    name: daemoneye-procmond
    namespace: daemoneye

ConfigMap and Secrets#

configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: daemoneye-config
  namespace: daemoneye
data:
  procmond.yaml: |
    app:
      scan_interval_ms: 30000
      batch_size: 1000
      log_level: info
      data_dir: /data
      log_dir: /logs
    database:
      path: /data/processes.db
      retention_days: 30
    security:
      enable_privilege_dropping: true
      drop_to_user: 1000
      drop_to_group: 1000
  daemoneye-agent.yaml: |
    app:
      scan_interval_ms: 30000
      batch_size: 1000
      log_level: info
      data_dir: /data
      log_dir: /logs
    database:
      path: /data/processes.db
      retention_days: 30
    alerting:
      enabled: true
      sinks:
        - type: syslog
          enabled: true
          facility: daemon
        - type: webhook
          enabled: true
          url: http://daemoneye-webhook:8080/webhook

secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: daemoneye-secrets
  namespace: daemoneye
type: Opaque
data:
  webhook-token: <base64-encoded-token>
  database-encryption-key: <base64-encoded-key>

Persistent Storage#

pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: daemoneye-data
  namespace: daemoneye
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: fast-ssd

DaemonSet for procmond#

procmond-daemonset.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: daemoneye-procmond
  namespace: daemoneye
spec:
  selector:
    matchLabels:
      app: daemoneye-procmond
  template:
    metadata:
      labels:
        app: daemoneye-procmond
    spec:
      serviceAccountName: daemoneye-procmond
      containers:
        - name: procmond
          image: daemoneye/procmond:1.0.0
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            runAsUser: 1000
            runAsGroup: 1000
          volumeMounts:
            - name: config
              mountPath: /config
              readOnly: true
            - name: data
              mountPath: /data
            - name: logs
              mountPath: /logs
          env:
            - name: DaemonEye_LOG_LEVEL
              value: info
            - name: DaemonEye_DATA_DIR
              value: /data
            - name: DaemonEye_LOG_DIR
              value: /logs
          command: [procmond]
          args: [--config, /config/procmond.yaml]
          resources:
            requests:
              memory: 256Mi
              cpu: 100m
            limits:
              memory: 512Mi
              cpu: 500m
          livenessProbe:
            exec:
              command: [procmond, health]
            initialDelaySeconds: 30
            periodSeconds: 30
          readinessProbe:
            exec:
              command: [procmond, health]
            initialDelaySeconds: 10
            periodSeconds: 10
      volumes:
        - name: config
          configMap:
            name: daemoneye-config
        - name: data
          persistentVolumeClaim:
            claimName: daemoneye-data
        - name: logs
          emptyDir: {}
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule

Deployment for daemoneye-agent#

daemoneye-agent-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: daemoneye-agent
  namespace: daemoneye
spec:
  replicas: 1
  selector:
    matchLabels:
      app: daemoneye-agent
  template:
    metadata:
      labels:
        app: daemoneye-agent
    spec:
      serviceAccountName: daemoneye-agent
      containers:
        - name: daemoneye-agent
          image: daemoneye/daemoneye-agent:1.0.0
          imagePullPolicy: IfNotPresent
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
          volumeMounts:
            - name: config
              mountPath: /config
              readOnly: true
            - name: data
              mountPath: /data
            - name: logs
              mountPath: /logs
          env:
            - name: DaemonEye_LOG_LEVEL
              value: info
            - name: DaemonEye_DATA_DIR
              value: /data
            - name: DaemonEye_LOG_DIR
              value: /logs
            - name: DaemonEye_PROCMOND_ENDPOINT
              value: tcp://daemoneye-procmond:8080
          command: [daemoneye-agent]
          args: [--config, /config/daemoneye-agent.yaml]
          resources:
            requests:
              memory: 512Mi
              cpu: 200m
            limits:
              memory: 1Gi
              cpu: 1000m
          livenessProbe:
            exec:
              command: [daemoneye-agent, health]
            initialDelaySeconds: 30
            periodSeconds: 30
          readinessProbe:
            exec:
              command: [daemoneye-agent, health]
            initialDelaySeconds: 10
            periodSeconds: 10
      volumes:
        - name: config
          configMap:
            name: daemoneye-config
        - name: data
          persistentVolumeClaim:
            claimName: daemoneye-data
        - name: logs
          emptyDir: {}

Service#

service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: daemoneye-agent
  namespace: daemoneye
spec:
  selector:
    app: daemoneye-agent
  ports:
    - name: http
      port: 8080
      targetPort: 8080
      protocol: TCP
  type: ClusterIP

Deploy Basic Setup#

# Create namespace
kubectl apply -f namespace.yaml

# Apply RBAC
kubectl apply -f rbac.yaml

# Apply configuration
kubectl apply -f configmap.yaml
kubectl apply -f secret.yaml

# Apply storage
kubectl apply -f pvc.yaml

# Deploy components
kubectl apply -f procmond-daemonset.yaml
kubectl apply -f daemoneye-agent-deployment.yaml
kubectl apply -f service.yaml

# Check deployment status
kubectl get pods -n daemoneye
kubectl get services -n daemoneye

Production Deployment#

Production Configuration#

production-configmap.yaml includes enhanced settings for scan intervals, database tuning, alerting with multiple sinks, detection rules with hot-reloading, observability with Prometheus metrics, and structured JSON logging.

Horizontal Pod Autoscaler#

hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: daemoneye-agent-hpa
  namespace: daemoneye
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: daemoneye-agent
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
    scaleUp:
      stabilizationWindowSeconds: 60

Helm Chart Deployment#

Helm Chart Structure#

daemoneye/
├── Chart.yaml
├── values.yaml
├── values-production.yaml
├── values-development.yaml
├── templates/
│ ├── namespace.yaml
│ ├── rbac.yaml
│ ├── configmap.yaml
│ ├── secret.yaml
│ ├── pvc.yaml
│ ├── procmond-daemonset.yaml
│ ├── daemoneye-agent-deployment.yaml
│ ├── service.yaml
│ ├── hpa.yaml
│ ├── networkpolicy.yaml
│ └── servicemonitor.yaml
└── charts/

Deploy with Helm#

# Add DaemonEye Helm repository
helm repo add daemoneye https://charts.daemoneye.com
helm repo update

# Install DaemonEye
helm install daemoneye daemoneye/daemoneye \
  --namespace daemoneye \
  --create-namespace \
  --values values.yaml

# Install with production values
helm install daemoneye daemoneye/daemoneye \
  --namespace daemoneye \
  --create-namespace \
  --values values-production.yaml

# Upgrade deployment
helm upgrade daemoneye daemoneye/daemoneye \
  --namespace daemoneye \
  --values values.yaml

# Uninstall
helm uninstall daemoneye --namespace daemoneye

Security Configuration#

Network Policies#

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: daemoneye-network-policy
  namespace: daemoneye
spec:
  podSelector:
    matchLabels:
      app: daemoneye
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: daemoneye
      ports:
        - protocol: TCP
          port: 8080
        - protocol: TCP
          port: 9090
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: daemoneye
      ports:
        - protocol: TCP
          port: 8080
    - to: []
      ports:
        - protocol: TCP
          port: 53
        - protocol: UDP
          port: 53

Monitoring and Observability#

Prometheus ServiceMonitor#

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: daemoneye
  namespace: daemoneye
spec:
  selector:
    matchLabels:
      app: daemoneye
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s
      scrapeTimeout: 10s

Troubleshooting#

Common Issues#

Pod Won't Start:

kubectl get pods -n daemoneye
kubectl logs -n daemoneye daemoneye-procmond-xxx
kubectl describe pod -n daemoneye daemoneye-procmond-xxx

Permission Denied:

kubectl get pod -n daemoneye daemoneye-procmond-xxx -o yaml | grep securityContext
kubectl exec -n daemoneye daemoneye-procmond-xxx -- ls -la /data

Network Issues:

kubectl get endpoints -n daemoneye
kubectl exec -n daemoneye daemoneye-agent-xxx -- ping daemoneye-procmond

Database Issues:

kubectl exec -n daemoneye daemoneye-agent-xxx -- daemoneye-cli database status
kubectl exec -n daemoneye daemoneye-agent-xxx -- daemoneye-cli database integrity-check

Performance Issues#

High CPU/Memory Usage:

kubectl top pods -n daemoneye
kubectl get hpa -n daemoneye
kubectl scale deployment daemoneye-agent --replicas=3 -n daemoneye

This Kubernetes deployment guide provides comprehensive instructions for deploying DaemonEye on Kubernetes. For additional help, consult the troubleshooting section or contact support.