Kubernetes does not differentiate sidecars and application containers in a Pod. Hence, enabling Istio for short-running workloads imposes additional challenges to the conventional approach of injecting an Envoy sidecar to the istio-enabled Pod.

• The proxy sidecar is long-running and prevents the Pod from finishing even after the application container has completed.
• We cannot ensure the proxy to be running and healthy before the application container starts. Without Istio proxy, requests to downstream services may fail if service authentication is enabled, and thus the app container would fail its health check, causing the Pod to be recreated. The pod may be stuck in a crash loop for a while.

Straw-man attempts

At first sight, it seems the initContainers in Pod is exactly what we wanted. After all, don’t initContainers always run before containers in Pod? Yet, no containers start until all initContainers have exited successfully. If we run Istio proxy in initContainers, then we have to kill it manually before the app container could start, which requires the proxy to connect to the service mesh.

Some popular mitigation I have seen is to change the entrypoint of the app container to wait a few seconds before executing the app binary. But how long of a wait is enough? Starting too soon risks proxy not healthy, but starting too late is a waste of time and resources. Scheduling is never deterministic. Worse, one will have to update the Pod spec themselves, which is not amenable to all the workloads in the cluster.

An Automated and Robust Solution

Start Pod: Mutating Webhook + Command Overwrites

You may build a binary (say, valet) that polls the /ready endpoint of the proxy, and then exec the app binary afterwards. We can use the mutating webhook from Kubernetes to automatically rewrite the container command to include valet. The valet binary can be downloaded using a init container and copied to the app container using a shared volume. The following Kubernetes manifest shows this approach.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
apiVersion: batch/v1
kind: Job
metadata:
  name: short-lived
spec:
  completions: 1
  parallelism: 1
  template:
    metadata:
      name: short-lived
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: short-lived
        myapp.com/valet: "true"
    spec:
      initContainers:
      - name: valet-init
        image: myapp/valet:latest
        command:
        - /bin/sh
        args:
        - -c
        - cp /valet /shared/valet
        volumeMounts:
        - mountPath: /shared
          name: shared
      containers:
      - name: short-lived
        image: myapp:latest
        command:
        - "/shared/valet"
        - "myapp"
        volumeMounts:
        - name: shared
          mountPath: /shared
      volumes:
      - name: shared
        emptyDir: {}

Finish Pod: Pod Exec + Kill Proxy

The valet binary also supports a -stop-proxy flag, which will stop Envoy proxy by calling the /quitquitquit endpoint on the pilot-agent. In addition to that, the valet controller will be watching all the pods that are labeled as myapp.com/valet: "true", and it will issue a valet -stop-proxy command when the short-lived containers are all completed. It will essentially be the below call.

1
kubectl exec -it $pod -c short-lived -- /shared/valet -stop-proxy