Hangar reached AWS with two access keys: passwords for machines that worked from anywhere until someone deleted them, one of them shared by every app. I wanted them gone, not rotated. Doing that also turned up a way for Crossplane to make itself an AWS administrator, and the switch itself deleted a function before I fixed the composition that allowed it. This page explains all three for someone meeting these ideas for the first time.
Hangar uses AWS in two ways. Crossplane builds what an app needs there: an image registry, a Lambda function, a public web address for it. Glidepath, the delivery pipeline, then ships each new version of the app's code into that function. Each needed an AWS identity, and each got an access key: a username and password for a machine.
A key like that has three problems. It never expires, so a leaked copy works until someone notices. It works from anywhere, so a copy on a laptop is as good as the original. And the deploy key was shared by every app, so a leak from one app's pipeline reached every other app's functions. Rotating keys shortens the window; it doesn't close it.
Think of a building with a front desk. The old way handed each contractor a master key to keep. The new way gives them nothing to keep. They arrive with a badge their employer printed that morning, the desk checks it against the employer's published sample, and hands back a visitor pass for one floor that expires in an hour.
The employer is the cluster: Kubernetes gives every pod a signed identity card, a ServiceAccount token.
The published sample is kiac-dev's public signing keys, at hangarplatform.dev/oidc/kiac-dev: anyone
can check a signature with them, nobody can make one. The front desk is AWS STS, which checks the token
against an IAM role's rules about who may use it, and returns credentials that last an hour. The industry
name is web identity federation; on Amazon's own Kubernetes service it's called IRSA.
| Who | Runs as | Assumes | Can do |
|---|---|---|---|
| Crossplane's AWS providers | provider-aws-* | hangar-crossplane-kiac-dev | create, observe and delete what the compositions describe |
| Glidepath's deploy step | glidepath-cloud-deployer | glidepath-deploy-<fn> | push to this function's registry, update this function's code |
| The target's bootstrap Job | glidepath-cloud-deployer | glidepath-deploy-<fn> | the same, once, to seed the first image |
| Every other pipeline step | pipeline-runner | nothing | no AWS access |
The Lambda target already creates the function, its registry and its execution role, so it creates the deploy role beside them and deletes it with them. A new app needs no IAM step at all.
Every pipeline step runs as pipeline-runner, including tests that run the app's own code. Only the deploy step and the bootstrap Job run as glidepath-cloud-deployer, the one subject the role trusts.
The AWS account lives in the fleet's clusters.yaml. The compositions and every app's chart read it from there; no app repeats it in cicd.yaml.
| Option | Verdict | Why |
|---|---|---|
| Web identity federation | Chosen | Nothing stored, nothing new in the request path, one role per function, and the same pattern every managed Kubernetes uses. |
| Rotate the keys | Rejected | Shortens the window but keeps it, adds a process someone must run, and keeps one key for every app. |
| Short-lived keys from a secrets store | Fallback | Real leases, but the store keeps a standing credential that can mint them, and every deploy depends on it being up. |
| IAM Roles Anywhere | Rejected | Trusts X.509 certificates instead of tokens: a certificate authority to run and certificates to hand out, for the same result. |
Trust pipeline-runner | Rejected | Simpler, but any step that runs the app's code could reach AWS. |
Its policy let it create any role ending in -exec, decide who may use it and attach any rules. Together that's a way to create a full administrator. Apps couldn't reach it, but a bug in a composition could.
A permissions boundary caps a role whatever its own rules say. Crossplane's policy now refuses to create a role, or add rules to one, unless the boundary is attached, and nothing may remove it.
Kubernetes lets any pod use any ServiceAccount in its namespace, and labels on a pipeline run are copied onto its pods. So the admission policies decide by who created the pod, which Kubernetes records and a client can't fake.
Switching every app at once would break any deploy that runs before its role exists, so for the switch the deploy step kept the old key as a fallback and logged which path it took. Once every app had deployed through its role and no cluster held an AWS key any more, the fallback was removed (phase 4). Now a deploy either assumes its role or stops and says what is missing.
Permissions boundary and Crossplane policy v3; every composed role carries the boundary.
The Lambda target creates glidepath-deploy-<fn>; its bootstrap Job uses it.
The ServiceAccount, the account ConfigMap, the deploy step and promote flows.
Four admission policies, 16 of 16 live cases. The first real deploy showed they also refused Tekton's signing and history services, which only add a note to a run; fixed the same day.
The fallback and the scaffold's AWS secrets are gone, and a test fails the build if the deploy step ever reads a stored key again. The old Infisical keys and IAM users are deleted by hand.
Since Oct 16, 2026 the ECS target composes an ECR repository and a deploy role too, and the deploy assumes it with a ServiceAccount token and copies the image into ECR. No stored key, and no app-secrets mounted.
Container Apps still log in with a stored service principal secret. Next: workload identity federation.
Crossplane works by comparison: each time it looks at an app, it lists what should exist and deletes anything no longer on the list. Phase 1 made the bootstrap Job wait for the new deploy role. For one pass the role didn't exist, so the finished Job dropped off the list and was deleted. An older rule said the function exists only once the Job has succeeded, so on the next pass the function, its URL and its permission dropped off too, and were deleted in AWS. They came back six minutes later from the placeholder image, with a new web address.
The fix keeps anything that already exists on the list, whatever happens to the conditions it was created under. Two new tests replay both steps; run against the old composition, they fail the way production did.
A condition may decide when to create something. It must never decide whether to keep it.
| Check | Result |
|---|---|
| Crossplane's role | Assumed from a real provider token; the default ServiceAccount, a wrong audience and the Azure provider were refused. |
| A real create and delete | An empty registry created, Ready and deleted through the role; AWS confirmed it gone. |
| Permissions boundary | Added to the existing execution role and on both new deploy roles, which policy v3 only allows with the boundary. |
| Bootstrap Job | Ran as glidepath-cloud-deployer and logged in to the registry through the deploy role, with no key mounted. |
| Admission policies | 16 of 16 live cases, 12 for creating objects and 4 for changing them, each test object checked to contain what its case claims. |
| A real deploy | smoke-fn promoted to production (promote-prod-0-deploy-cc5kj): the deploy step ran as glidepath-cloud-deployer, logged credentials: assuming …/glidepath-deploy-glidepath-smoke-fn-prod (web identity, no stored key) and updated the function. |
| With no key path at all | After phase 4, the same promotion re-run (promote-prod-0-deploy-t4n84): the deploy step had no app-secrets volume, assumed the role, updated the function, and was signed and recorded. |
The page's numbers are from kiac-dev and AWS account 700017004810 on October 16, 2026. Pull requests: hangar#55, glidepath#114 to #116, airframe#87 to #90, gitops-cluster-dev#96 to #108, apron#30 and #31, and for phase 4 glidepath#118 and airframe#92 and #93; documentation in airframe#91, glidepath#117 and hangar#56.