Docs · Tower

Cluster credentials: how Tower reaches

Tower’s backend (Backstage, on ) reads and writes without any stored Kubernetes credential.

• ’s API server trusts ’s ServiceAccount token issuer.
• A token that issues for is accepted there.
• Reads use the Backstage pod’s own token, which the kubelet rotates.
• Writes use a token minted for one action and then dropped.
• Each of the two identities gets only its own permissions, and admission policies check every write.

This page explains the design, the configuration, and how to operate, verify and troubleshoot it. The background and the rollout record are in ~/tech/SPEC-tower-cluster-credentials.md. The cluster-side mechanism is generic and lives in apron (docs/cluster-identity.md, components.clusterIdentity).

Until 2026-10-12 Tower used one kubernetes.io/service-account-token for dev.

• It never expired.
• It lived in four places: dev, Infisical, a prod Secret, and the process environment.
• It was sent without TLS verification.
• It could create PipelineRuns in any namespace, which made it cluster-admin-equivalent.
• Any signed-in user could use it through the Kubernetes proxy.

That token and everything that carried it are deleted.

The two identities

On Backstage is simply its in-cluster ServiceAccount (system:serviceaccount:backstage:backstage-ingestor), with its usual short-lived token. Nothing on this page changes that.

A write, end to end

1. The browser posts to the backend’s own route (/api/glidepath/tekton/rerun or /cancel; a Flight promotion or rollback: /promote, /release/rollback). Tower never writes through the Kubernetes proxy, which refuses anything but GET.
2. The backend checks the permission on the app’s catalog entity (tower.pipeline.rerun / .cancel, or tower.release.promote / .rollback: owners or admins). It audits the decision either way, and mints nothing for a refused request.
3. towerActor.ts asks for a tower-actor token. Backstage’s ServiceAccount may create tokens for that one name only (Role tower-actor-token-minter).
4. The backend calls glidepath-api with that token, through ’s API server (/api/v1/namespaces/platform-system/services/glidepath-api:8080/proxy/v1/...), naming the person in X-Glidepath-On-Behalf-Of (packages/backend/src/glidepathApi.ts).
5. ’s API server checks the token against ’s published keys and the claim rules (below). It maps the token to hangar:<prod-cluster>:tower-actor and applies that user’s RBAC, which opens glidepath-api’s service proxy and nothing else on Tekton.
6. glidepath-api checks the token again (TokenReview), authorizes tower-actor with a SubjectAccessReview on its own resources (rerun/cancel on runs.glidepath.hangar.io, promote/rollback on environments.glidepath.hangar.io, from the glidepath-api-operator and glidepath-api-releaser roles), audits the call with the person, and creates or patches the PipelineRun as its own ServiceAccount (system:serviceaccount:platform-system:glidepath-api). glidepath-api-pipelinerun-shape admits only Glidepath’s own run shape.

Until P6 (2026-10-30) the backend created and patched the PipelineRun itself as tower-actor.

Break-glass works the same way. It mints a fresh token for each step (activation, attach, state reads, the end of the session), because a session can outlive a 10-minute token. Its RoleBinding’s subject is the hangar:<prod-cluster>:tower-actor user, which the backend learns from a SelfSubjectReview.

Guardrails

What remains. Code running inside the Backstage backend can mint actor tokens. Those tokens can only call glidepath-api’s writes (re-run, cancel, promote, rollback), which glidepath-api authorizes and audits, and make break-glass’s debug grants, held to their shape by the policies. Each one is gated on ownership in the routes and recorded. Writing as each user’s own identity (spec phase 5) is what would narrow it further.

Configuration

Backstage: app-config.yaml

kubernetes:
  clusterLocatorMethods:
    - type: config
      clusters:
        - name: <dev-cluster>
          url: https://<dev-cluster>-control-plane:6443        # the node name, which is in the API server certificate
          authProvider: serviceAccount
          authMetadata:
            hangar.io/token-file: /var/run/secrets/hangar/<dev-cluster>/token   # reads
            hangar.io/actor-service-account: backstage/tower-actor         # writes, minted per action
            hangar.io/actor-audience: <dev-cluster>
            hangar.io/actor-issuer-cluster: <prod-cluster>                       # the entry that mints it
          serviceAccountToken: unused-the-token-file-is-used                # never sent; see below
          caData: <the dev cluster's CA, base64 PEM>
        - name: <prod-cluster>
          url: https://kubernetes.default.svc
          authProvider: serviceAccount
          caFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

• hangar.io/token-file. kubernetesAuthStrategies.ts replaces the serviceAccount strategy with one that reads this file on every request (and otherwise behaves like the stock strategy). It also re-registers every stock strategy, because adding any strategy replaces the plugin’s default map. clusterAccess.ts uses client-node’s tokenFile provider, which re-reads the file every 60 s.
• The three actor keys must be set together. towerActor.ts writeKubeConfig() uses them. Without them, writes fall back to the entry’s fixed token, which is the placeholder, so they are refused: the setup fails closed.
• The placeholder serviceAccountToken must stay. @backstage/plugin-kubernetes-backend’s proxy (0.21.10, KubernetesProxy.getClusterForRequest) treats a serviceAccount cluster with no serviceAccountToken as the pod’s own cluster when Backstage runs in-cluster, and rewrites its URL to kubernetes.default.svc. Removing the placeholder sends every kind-dev read to prod. That happened for 21 minutes on 2026-10-12, as 401s on every proxy read. A local harness can’t reproduce it, because it doesn’t run in-cluster.

: 60-backstage/backstage/

• deployment.yaml:
  • the dev-cluster-token projected volume (audience <dev-cluster>, expirationSeconds: 600), mounted at /var/run/secrets/hangar/<dev-cluster>;
  • the <dev-cluster>-control-plane hostAlias.
• tower-actor.yaml: the tower-actor ServiceAccount and the minter Role and RoleBinding.
• ’s API server signs tokens as https://<issuer-host>/oidc/<prod-cluster>. Its first --service-account-issuer is that URL; the default stays as a second, accepted issuer; and --api-audiences lists both (00-bootstrap/cluster-identity/README.md).

: gitops-cluster-dev

• 00-bootstrap/cluster-identity/authentication-config.yaml: the trust rules. It’s node-level, so Argo CD doesn’t sync it; it’s installed with apron hack/cluster-identity.sh trust.
• 00-bootstrap/backstage-ingestor-rbac/: the reader’s and the actor’s RBAC (rbac.yaml, tower-actor-rbac.yaml, breakglass-rbac.yaml). The reader’s glidepath-api grant keeps its old Role name, backstage-ingestor-tekton-results-proxy, so the Application (no prune) updates it in place.
• 30-policy/kyverno-policies/: tower-identity-writes.yaml, tower-breakglass.yaml, no-long-lived-sa-tokens.yaml.

The issuer’s public keys

The keys are served by hangar’s site at https://<issuer-host>/oidc/<prod-cluster>/.well-known/openid-configuration and /jwks.json. They’re public keys only; the source is the issuer site’s oidc/<prod-cluster>/.

Operations

Verify the trust end to end. This is read-only: it mints a token on and presents it to with no other credential.

apron/hack/cluster-identity.sh verify --from-context <prod-cluster> --namespace backstage \
  --service-account backstage-ingestor --audience <dev-cluster> --to-context <dev-cluster>
# accepted as hangar:<prod-cluster>:backstage-ingestor; wrong audience, 20-minute token, backstage/default refused

Is Backstage authenticating? counts JWT authentications per issuer. apiserver_authentication_jwt_authenticator_latency_seconds_count{result="success"} keeps rising while Backstage reads:

kubectl --context <dev-cluster> get --raw /metrics | grep '^apiserver_authentication_jwt_authenticator_latency_seconds_count'

Kill switch. Edit /etc/kubernetes/authn/config.yaml on ’s control-plane node and delete the jwt entry. The API server reloads the file within seconds, and every token stops working, readers and actors alike. To restore it, run hack/cluster-identity.sh trust again with the file from gitops-cluster-dev.

Narrower revocation.

• Remove a user’s bindings in gitops-cluster-dev.
• Or remove tower-actor from the trust file’s ServiceAccount list, which stops writes and keeps reads.
• Or drop the minter Role on .

If ’s signing key changes (a recreated cluster, or a rotated sa.key), republish the keys before anything else, or refuses every token:

apron/hack/cluster-identity.sh publish --context <prod-cluster> --issuer https://<issuer-host>/oidc/<prod-cluster> \
  --out <issuer-site>/oidc/<prod-cluster>

A recreated also needs its issuer flags again (apron components.clusterIdentity / hack/cluster-identity.sh issuer). A recreated needs the trust file again (trust), plus its new CA in the <dev-cluster> entry’s caData.

Local harness. The old ServiceAccount is gone. Impersonate the federated users instead:

kubectl --context <dev-cluster> --as=hangar:<prod-cluster>:tower-actor --as-group=hangar:<prod-cluster> proxy --port=8013 --reject-paths='^/never$'

tower-actor can read (view), call glidepath-api and bind the debug roles; it can’t create or patch a PipelineRun. Use hangar:<prod-cluster>:backstage-ingestor to check that something works read-only.

Troubleshooting