Break-glass · 05 of 07 · The policies
Six admission policies, one per thing that could go wrong
The backend is the first lock. These are the second, inside each cluster's API server, written so that a bug in Tower, or a compromised Backstage, still cannot turn a debugging session into something bigger.
Six admission policies, one per thing that could go wrong
Top: what Backstage asks the API server to do: create a RoleBinding (the grant), add a debug container (ephemeral, or in a copy pod), delete a pod (cleanup), patch a pod (process access); and, from anyone on Flight, exec or attach. Below, six policies named tower-breakglass-something, all failing closed. rolebinding: the name starts tower-breakglass-, the role is debug-live or debug-copy, the only subject is Backstage, a TTL label of 15 to 75 minutes, an app namespace only, and on Flight either an approver who is not the requester or the glass-broken mark, never neither. debug-container: the pinned image digest, runAsNonRoot with a non-zero group, a read-only root, no privilege escalation, all capabilities dropped, every mount read-only, and process access only for the session the pod's annotation names. debug-image: a cosign keyless signature from tower's own workflow on main, logged in the public Rekor; it runs in Kyverno as a webhook scoped to these requests and fails closed too. pod-delete: Backstage may delete only copy pods or pods carrying a debug container, nothing else in the namespace. pod-annotate: the spec and labels are unchanged and only the process-access annotation with the session id may change. no-exec: no exec into app pods, attach only to debug containers, operators excepted, Flight only. Bottom: on Ground they are Kyverno ValidatingPolicies compiled to native admission policies with no webhook; on Flight they are hand-written ValidatingAdmissionPolicies synced by Argo CD before the backend that needs them.
WHAT BACKSTAGE ASKS THE API SERVER TO DO TOWER-BREAKGLASS-* · EVERY ONE FAILS CLOSED CREATE RoleBinding the grant Add a debug container ephemeral, or a copy pod DELETE a pod cleanup PATCH a pod process access exec or attach anyone, on Flight rolebinding name tower-breakglass-* role: debug-live, -copy subject: Backstage TTL label 15m to 75m app namespace only Flight: approver is not the requester, or glass broken, never neither debug-container image: pinned digest runAsNonRoot, gid not 0 read-only root no privilege escalation capabilities: drop ALL every mount read-only process access only for the session named on it debug-image cosign keyless signed from tower's workflow on main, public Rekor Kyverno, as a webhook scoped to break-glass fails closed too pod-delete deletes copy pods, or pods with a debug-* container, only nothing else in the namespace pod-annotate spec unchanged labels unchanged one annotation may change: process-access with the session id no-exec no exec into app pods attach: debug-* only operators excepted Flight only Ground: Kyverno ValidatingPolicy compiled to native admission policy · no webhook Flight: hand-written ValidatingAdmissionPolicy Argo CD · synced before the backend that needs it Every rule is scoped to break-glass objects, so a policy bug can only ever break break-glass. LEGEND A request Backstage makes A request anyone makes Policy How it is deployed
Validation runs as built-in ValidatingAdmissionPolicy, inside the API server. It cannot be down on its own, which matters most during the incident you opened the session for.
Only the image signature needs Kyverno's webhook. Its match conditions are copied onto the webhook, so it never sees a request that is not break-glass.
An unguarded CEL field once refused every pod create on the dev cluster for ten minutes. Now every optional field is guarded and every policy is scoped.