Wiki

* Scroll me *
PREFACE
About Wiki
GOLANG
Snippets
Pitfalls
Development
Production
SHELL
Shell Script
NETWORKING
Networking
KUBERNETES
Commands
Patterns
Potholes
ISTIO
Istio
VAULT
Vault
INFRA AS CODE
Terraform
Pulumi

Kubernetes Patterns

Table of Contents

Run-once DaemonSet Pattern

Some Kubernetes providers, such as GKE and AKS, do not support custom node images. The run-once DaemonSet pattern is useful to prime worker nodes for security hardening or image prefetch. Regular DaemonSet does not work since they restart the Pod when the setup task is done. We want the DaemonSet to run once.

We can do so by running the setup script in the init container, and run the pause container afterwards, which consumes little resources.

Below is an example of run-once DaemonSet to implement the GKE CIS benchmark recommendation of enalbing --protect-kernel-defaults on the kubelet.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cis-protect-kernel-defaults
  namespace: kube-system
  labels:
    app: protect-kernel-defaults
spec:
  selector:
    matchLabels:
      name: protect-kernel-defaults
  template:
    metadata:
      labels:
        name: protect-kernel-defaults
    spec:
      hostNetwork: true
      hostPID: true
      hostIPC: true
      volumes:
        - name: root-mount
          hostPath:
            path: /

      initContainers:
        - image: alpine:3.18
          name: protect-kernel-defaults
          command: ["/bin/sh", "-c"]
          # The advantage of args over mounting a configmap is that
          # when the script is updated, kubernetes will rolling
          # update Pods.
          # By using chroot, you can run commands as if you were
          # executing them directly on the node, not just inside
          # a container.
          args:
            - |
              ROOT_MOUNT_DIR="/node_root"
              KUBELET_CONFIG_FILE="/home/kubernetes/kubelet-config.yaml"
              WANT_SETTING="protectKernelDefaults: true"

              echo "Current kubelet config:"
              cat "${ROOT_MOUNT_DIR}/${KUBELET_CONFIG_FILE}"

              # This check ensures `protectKernelDefaults' is defined once.
              # It is invalid YAML to have duplicated keys.
              found=$(grep "${WANT_SETTING}" "${ROOT_MOUNT_DIR}/${KUBELET_CONFIG_FILE}")
              if [[ ! -z "${found}" ]]; then
                echo "protectKernelDefaults is already set to true. Done."
                exit 0
              fi

              echo "Updating kubelet config file."
              echo "${WANT_SETTING}" >> "${ROOT_MOUNT_DIR}/${KUBELET_CONFIG_FILE}"

              echo "Restarting kubelet."
              chroot "${ROOT_MOUNT_DIR}" systemctl restart kubelet

              echo "Done."
          securityContext:
            privileged: true
          volumeMounts:
            - name: root-mount
              mountPath: /node_root

      containers:
        - name: pause
          image: gcr.io/google_containers/pause:3.2

Best practices

See How to Configure Applications for High Availability in Kubernetes