Wiki

* Scroll me *
PREFACE
About Wiki
GOLANG
Snippets
Pitfalls
Development
Production
SHELL
Shell Script
NETWORKING
Networking
KUBERNETES
Commands
Patterns
Potholes
ISTIO
Istio
VAULT
Vault
INFRA AS CODE
Terraform
Pulumi

Kubernetes Potholes

Table of Contents

Webhook Blocking Webhook Deployment Itself

If your MutatingWebhookConfiguration has failurePolicy: Fail, then make sure to configure the webhook to ignore the webhook deployment itself, such as using a namespace selector.

I hit this issue with vault-agent-injector. The webhook “vault.hashicorp.com” is backed by vault-agent-injector-svc, which requires deployment/vault-agent-injector to be up to serve the webhook. When vault-agent-injector has 0 replica, the “vault.hashicorp.com” webhook prevents vault-agent-injector itself from scaling up despite HPA min=2.

Because vault-agent-injector is down and the webhook is down, GKE netd daemonset pods cannot run on new nodes to configure CNI on these nodes. New nodes are all NotReady, because runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized.

This will cause any new nodes created as part of cluster autoscaling to be unhealthy, and will risk cascading failure of the cluster.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

❯ kubectl get deploy -n vault
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
vault-agent-injector   0/2     0            0           193d

❯ kubectl describe deploy -n vault
Name:                   vault-agent-injector
Namespace:              vault
...
OldReplicaSets:    <none>
NewReplicaSet:     vault-agent-injector-69988955f5 (0/2 replicas created)

❯ kubectl describe rs vault-agent-injector-69988955f5 -n vault
...
Events:
  Type     Reason        Age                   From                   Message
  ----     ------        ----                  ----                   -------
  Warning  FailedCreate  15m (x41 over 5h49m)  replicaset-controller  Error creating: Internal error occurred: failed calling webhook "vault.hashicorp.com": Post "https://vault-agent-injector-svc.vault.svc:443/mutate?timeout=10s": no endpoints available for service "vault-agent-injector-svc"

❯ kubectl describe ds netd -n kube-system
...
Events:
  Type     Reason        Age                     From                  Message
  ----     ------        ----                    ----                  -------
  Warning  FailedCreate  7m18s (x42 over 4h39m)  daemonset-controller  Error creating: Internal error occurred: failed calling webhook "vault.hashicorp.com": Post "https://vault-agent-injector-svc.vault.svc:443/mutate?timeout=10s": no endpoints available for service "vault-agent-injector-svc"