Tower · Spec · Break-glass debug access

Looking inside a production pod, without ever handing out exec

Tower's write actions are delegated and reviewed, so pod exec was never going to be a standing feature. But mid-incident an owner sometimes needs a process list, a DNS lookup from the pod's network, or a file the app wrote, and no log line answers it. This is the spec for doing that safely: short, approved, recorded, and enforced by the cluster itself.

Spec October 10, 2026Status phases 1 to 4 built and live on both clusters, Oct 12, 2026Test plan October 12, 2026Docs Tower break-glass guide
Built · live on both clusters

Never exec on Flight. Two debug modes, a second person, and a recording of every byte.

Exec or a debug container?

exec into the appephemeral debug container
Admission controlallow or deny the whole thingthe policy sees the full container spec
What runsthe app image's tools, as the app's useran approved, pinned image as a fixed non-root user
Writesland in the app containerimpossible: read-only root, read-only mounts
Minimal imagesoften no shell at allalways works
Trace in the podnonethe container stays in the pod spec

A debug container is safer for these reasons, not inherently. Run as the app's user with a shared process namespace, it reaches the same secrets as exec. So process access is a separate, explicit option that the approver sees.

The flow

The break-glass flowAn owner requests a session with a reason, a second person approves, Tower creates a time-limited RoleBinding and attaches to a pinned debug container, the session is recorded, the pod is recycled at the end, and the record is reviewed.TOWERRequestowner, reasonApproveanother personSessionrecordedEndpod recycledReviewrecord and hashTHE CLUSTERRoleBindingTTL, deleted at the endDebug containerpinned, signed imageACTIVATEATTACHAdmission policiescheck every step
Figure 1Outside an active grant, Tower holds none of the verbs a session needs. The RoleBinding carries a TTL, so it expires even if Tower dies mid-session.

Guardrails that don't trust the portal

Four layers of guardrailsTower permissions, the Tower backend, admission policies in the cluster, and time-limited RBAC.Tower permissionswho may request, approve, break the glass, endTower backendrequester is never approver; refuses an unrecorded startAdmission policiesbinding shape, debug image, no exec on FlightRBAC with a TTLnothing standing; the grant expires on its own
Figure 2Each layer assumes the one above it can fail. On Flight the validation runs as built-in admission policy inside the API server, so it's never down on its own.

The first build used fail-closed Kyverno policies on Flight too. On the dev cluster they made break-glass unavailable every time Kyverno lost its lease on a busy node, and on Flight that would happen during the incident itself. So Flight validation moved to built-in ValidatingAdmissionPolicy, and a slim Kyverno keeps only the two jobs it's uniquely good at: the cleanup TTL and the debug image's signature check.

How it was tested

The last piece is a plan for walking every feature in a browser with two people, one an admin and one an app owner, on a Skyport app's Ground and Flight environments. It covers the things a harness can't: the approval round trip, the notification deep links, a sensitive recording refused to the wrong person, and a "glass broken" session reviewed by someone else.

Ground: copy, held copy, live with process access

Plan

Non-root, no token, read-only; the recording's hash matches and is anchored in the Rekor transparency log.

Flight with two people

Plan

Request, approve with "keep the pod", deny, end by the approver, and exec refused.

Glass broken, then reviewed

Partly verified Oct 12, 2026

Broadcast to everyone, review due, never self-reviewed, and a fallback reviewer.

Recordings

Verified Oct 12, 2026

A locked bucket refuses deletes, and the hash checks against the log without Tower.

Cluster backstops

Plan

The TTL deletes a binding at fifteen minutes; process access is refused unless the session asked for it.

Still open