Docs · Tower

Admission policies

Backstage’s own checks are the first lock. These policies are the second, inside each cluster’s API server, so that a bug in the backend, or a compromised Backstage, still cannot widen what a session may do. They all fail closed.

Ground writes them as Kyverno ValidatingPolicy (gitops-cluster-dev 30-policy/kyverno-policies/tower-breakglass.yaml) with autogen.validatingAdmissionPolicy.enabled: true. Kyverno compiles each into a native ValidatingAdmissionPolicy, vpol-<name>, and registers no webhook, so the API server enforces it even while Kyverno is down. (They were ClusterPolicy webhooks until 2026-10-11; Kyverno 1.19 deprecates ClusterPolicy, and the webhooks made break-glass unavailable whenever Kyverno lost its lease on a busy node.)

Flight writes them directly as ValidatingAdmissionPolicy + binding (gitops-cluster- 30-policy/breakglass/policies.yaml, Argo CD app tower-breakglass, sync wave -1).

tower-breakglass-rolebinding

Matches CREATE and UPDATE of rolebindings where the name starts with tower-breakglass-, or the writer is Backstage. Every other RoleBinding write skips it.

Kubernetes’ own RBAC escalation check adds one more: Backstage can bind only roles it is allowed to bind, which the binder role limits to the two debug roles by resourceNames.

tower-breakglass-debug-container

Matches UPDATE of pods/ephemeralcontainers, and CREATE of pods labelled hangar.io/breakglass-copy. Every other pod create skips it.

Every debug container (an ephemeral container, or a debug-* container in a copy pod) must have:

Copy pods additionally must carry a cleanup.kyverno.io/ttl label of 15m to 75m (this rule applies to admins too, so even an admin’s copy expires). On Flight a copy pod also may not use hostNetwork, hostPID or hostIPC.

Process access (G14). An ephemeral container this update adds with targetContainerName is admitted only if the pod carries hangar.io/breakglass-process-access: <session id> and the container is named debug-<that id>. The backend writes that annotation, under tower-breakglass-pod-annotate below, only for an approved request that had process access; so a backend bug (or a hand-made request) cannot add process access the approver never saw. Containers already in the pod are not re-judged.

Ground exempts system:masters from the container rules (an operator’s own kubectl debug) and applies them only in hangar.io/app namespaces. Flight applies them to everyone, in every namespace: an operator’s own kubectl debug on must also use the Tower debug image, hardened. --profile=restricted alone does not set a group or a read-only root, so add them:

cat > debug-custom.json <<'EOF'
{"securityContext": {"runAsUser": 65532, "runAsGroup": 65532, "runAsNonRoot": true, "allowPrivilegeEscalation": false,
  "readOnlyRootFilesystem": true, "capabilities": {"drop": ["ALL"]}}}
EOF
kubectl debug -it <pod> -n <ns> --image=ghcr.io/<owner>/tower-debug@sha256:<digest> \
  --profile=restricted --custom=debug-custom.json -- sh

tower-breakglass-pod-delete

Matches DELETE of pods by Backstage only.

Both debug roles include delete pods for the length of a session (copy mode deletes its copy; live mode restarts a pod Argo CD does not manage). Without this policy, a session’s grant would reach every pod in the namespace. The rule: the pod being deleted (oldObject) is a copy pod (hangar.io/breakglass-copy: "true") or carries an ephemeral container named debug-*.

tower-breakglass-pod-annotate

Matches UPDATE of pods (the main resource, not a subresource) by Backstage only.

Backstage’s live role includes patch pods for the length of a session, used for one thing: the process-access annotation above. This policy makes sure that is all the patch can do: object.spec == oldObject.spec, the labels are unchanged, and the annotations differ in no key other than hangar.io/breakglass-process-access (added, changed or removed). Tested in isolation on : the annotation admitted; a label, an image or another annotation refused.

tower-breakglass-debug-image

Matches the same requests as tower-breakglass-debug-container (an ephemeral-container update, or a copy pod), and only looks at images under ghcr.io/<owner>/tower-debug.

An ImageValidatingPolicy: the image must carry the cosign keyless signature of tower’s debug-image.yml workflow on main (issuer https://token.actions.githubusercontent.com, the public Fulcio and Rekor). The digest pin in tower-breakglass-debug-container says which build; this says the build came from that workflow. Image verification needs Kyverno’s admission controller, so this is the one break-glass policy that runs as a webhook, and it fails closed; it is therefore scoped: Kyverno copies the policy’s matchConditions onto the fine-grained webhook pair it registers for it (validating and mutating: ivpol.*.kyverno.svc-fail-finegrained-tower-breakglass-debug-image), so the webhook sees nothing but those two kinds of request. Check it after any change: kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml | grep -B2 -A6 finegrained-tower. (The CRD has no webhookConfiguration.matchConditions; setting one is pruned and leaves Argo CD OutOfSync.) Verified on both clusters: a plain pod create is untouched; a copy pod with the signed digest is admitted; a tag under the debug repository that does not exist is refused.

On Flight this is why runs a slim Kyverno (30-policy/kyverno: admission and cleanup controllers only, no background or reports controller): this policy, and the cleanup controller’s TTL backstop (30-policy/kyverno-policies/tower-breakglass-rbac.yaml gives it delete on RoleBindings and pods). Validation stays in the hand-written VAPs; if Kyverno is down on , a session can still be refused by it, never admitted unchecked.

tower-breakglass-no-exec (Flight)

Matches CONNECT on pods/exec and pods/attach in namespaces labelled hangar.io/app, for anyone not in kubeadm:cluster-admins or system:masters.

This makes “never exec on Flight” a cluster fact, not a Tower convention. People keep kubectl logs, port-forward and everything else their RBAC allows.

Writing CEL that fails closed safely

With failurePolicy: Fail, a CEL evaluation error refuses the request. In a match condition that is scoped too widely, one error refuses every request the policy matches, cluster-wide.

This happened on dev on 2026-10-11: a match condition read request.subResource unguarded. On a plain pod create that field is absent (not empty), the expression errored, and every pod create in the cluster was refused for about ten minutes. The rules that came out of it:

1. Guard every optional field with has() before reading it, in match conditions, variables and validations: has(request.subResource) && request.subResource == 'ephemeralcontainers'. That includes object.metadata.labels, .annotations, request.userInfo.username, object.subjects, c.securityContext and its children.
2. Scope each policy with match conditions to break-glass objects only (name prefix, label, or Backstage’s username), so an error can only ever affect break-glass.
3. On DELETE, the object is oldObject; object is null.
4. A Kyverno policy on Pods alone also autogenerates controller variants (Deployment, Job, …) and is then served from a webhook instead of a VAP. tower-breakglass-pod-delete sets autogen.podControllers.controllers: [].

Changing a policy without an outage

1. Test the exact CEL first, on , in a throwaway policy bound to a scratch namespace (see Testing). Cover every allow and every deny.

2. Widen before you narrow. When a value changes (the image digest, the TTL list), first make the policy accept both old and new, deploy, then switch the backend, then drop the old value. ’s debug-container policy keeps the allowed images in a list variable (variables.images) for this. Doing it in the other order refuses every session in between.

3. Order on Flight: the policy PR merges and syncs before the backstage bump that relies on it. Check that the live object changed, not just that Argo CD says Synced (Argo CD can report Synced against the previous revision):

   kubectl get validatingadmissionpolicy tower-breakglass-rolebinding -o yaml | grep <new text>
   kubectl -n argocd get app tower-breakglass -o jsonpath='{.status.sync.revision}'

4. Dry-run against the live policy as Backstage after it syncs (see Testing).