Hangar Platform · Airframe · Developer Guide
A hands-on walkthrough that ends with a running application: create a
service, get its pipeline running, pull in the code, give it a ground environment
to iterate in and a flight environment to run in, add a Redis cache, and finish
with a real canary rollout you can watch. We build boarding-api, the
gate board of Skyport: a small NodeJS service that caches gate lookups in Redis
and has a page built to make canary and blue/green deployments visible.
What has and hasn't been verified
Verified: the app (11 tests, and it ran against a real Redis with two instances
sharing one counter). Verified live: create → onboard → ground, and the
Redis component: provisioned through the catalog on both clusters, the gate board
reports cache: redis, and the password is read straight from the component's
Secret. Walking this guide the first time found and fixed real problems (the Redis image,
the devCluster name, the labels on the cache's resources), which is why the
text names them. Not verified: the canary in section 09.
Two architectures
The dev cluster is arm64 and the prod cluster is amd64.
Leave build.platforms unset in cicd.yaml so the image is built
for both; narrow it and the other cluster gets exec format error.
01 — GROUND & FLIGHT
Every Airframe app can have two kinds of environment, with different owners and different review bars. Hangar calls them ground and flight.
Ground
platform/envs/<name>.yaml in your source repodeploy stage, committed straight to
main — no PRplatform/envs/<name>.yaml by handFlight
ApplicationEnvironment XR, rendering
gitops-<app>/<cluster>/<env>/values.yamlrelease stage, as a reviewed pull requesttype: upperBoth render through the same Helm chart,
airframe-application. "Ground" and "flight" name the tier, not a
field value — you'll still type real names like dev and
staging below. Here is the whole path:
02 — CREATE THE APP
A gate screen looks up a flight, boarding-api returns the gate and boarding
info, and caches the answer for three minutes so a busy screen doesn't hammer the
reservations system. Its page also shows which version answered, which is what
makes the canary in section 09 visible.
Every Bootstrap-tier stack — NodeJS, Spring Boot, Go, Python, InfraService — is a Backstage Scaffolder template generated from its XRD. Fill in:
| Field | Value |
|---|---|
| Name | boarding-api |
| devCluster | dev |
| description | Skyport gate board: Redis-cached flight lookups and a canary visualizer |
| nodeVersion | 20 |
| packageManager | npm |
| port | 8080 |
| visibility | private |
Submitting opens a PR into gitops-cluster-dev-tenants at
tenants/boarding-api/xr-requests/boarding-api.yaml.
Merge it — the xr-requests ApplicationSet applies that file to
the dev cluster as a NodeJSApplication XR.
$ kubectl get nodejsapplication boarding-api -n app-boarding-api-cicd NAME SYNCED READY COMPOSITION AGE boarding-api True True nodejsapplications.catalog.idp.io 3m
kubectl describe shows the two custom conditions this
catalog adds: DevClusterReady and CicdOnboarded (Glidepath
has picked the app up).
Crossplane creates the real boarding-api source repo and an
empty gitops-boarding-api repo. The source repo starts with a hello-world
index.js, a package.json, a Containerfile and a
minimal cicd.yaml (build only, agent: nodejs-20,
unit tests off). You have a repo — but no running pipeline, and nothing deployed.
Use dev exactly
Type dev for devCluster, even if your dev cluster goes by another
name. The dev ApplicationSets hard-code cluster: dev when
they render an environment's ExternalSecret, so it reads the store
<app>-dev. An app created with any other value gets a store
named <app>-<that value>, its ExternalSecret never syncs, and its pods sit in
CreateContainerConfigError. And never remove dev from the
cluster-registry: every existing app is pinned to it, and removing it turns their stores
InvalidProviderConfig fleet-wide.
Starting over?
If a previous boarding-api was decommissioned, finish that first: delete its
Infisical projects (boarding-api-dev, boarding-api-prod)
and any boarding-api key in the prod cluster's secretstore-provisioner
ConfigMap, or the new app tries to adopt a dead project. Move any old local
boarding-api checkout out of the way too — section 04 clones into that name.
03 — PIPELINE ONBOARDING
Pipelines run through Pipelines-as-Code, which reads pipeline definitions from a
.tekton/ folder in the source repo. You don't write those files —
Glidepath does — but they have to get into your repo, and that takes one pull
request that you merge.
Once the app is registered, Glidepath's onboarding hook notices your
repo has no .tekton/onboarding-resync.yaml and opens a PR against
boarding-api (and gitops-boarding-api) adding the
generated .tekton/*.yaml files. Find it in GitHub or in Tower's
Pull Requests tab.
Until it's merged, pushing to main does nothing — PaC
finds no pipeline to run. Once merged, a push runs the scaffolded
build stage and publishes a multi-arch image to
ghcr.io/<owner>/boarding-api.
These files are never hand-edited. Every later push that changes
cicd.yaml opens a fresh PR regenerating .tekton/ if
anything changed — merge those too. If nothing happens after you push, first check
that the platform's GitHub App has access to the repo.
04 — PULL IN THE CODE
The real boarding-api lives in the airframe repo under
examples/skyport/boarding-api/. Copy it over the scaffold.
git clone https://github.com/jfillman/airframe.git /tmp/airframe
git clone https://github.com/jfillman/boarding-api.git && cd boarding-api # fails if ./boarding-api exists
cp -R /tmp/airframe/examples/skyport/boarding-api/. .
git add -A && git commit -m "boarding-api: gate board with Redis-backed lookups"
This overwrites index.js and package.json and adds
app.js, store.js, reservations.js, public/,
test/ and test.sh. Keep the scaffolded
Containerfile and, for now, cicd.yaml — you change it in
section 06 and push then. Try it locally first:
npm install && npm test # 6 passing PORT=8080 node index.js # open http://localhost:8080
With no REDIS_URL the header says cache: memory: the
app falls back to an in-process store so it runs anywhere. With two replicas each pod keeps
its own counts — the exact problem Redis fixes in section 08.
| Endpoint | Purpose |
|---|---|
GET / | The gate board: flight lookup, boarding-pass scans, version tally |
GET /api/boarding/:flight | Boarding info for a flight like AC123; cached 3 minutes; reports cached and latencyMs |
POST /api/boarding/:flight/scan | Increments that flight's boarded counter |
GET /api/whoami | Version, pod and cache mode — what the canary tally polls |
GET /healthz | Liveness / readiness |
The cache-aside lookup at the heart of it, from
app.js: check the cache, fall back to the slow path on a miss, set a short TTL
so a gate change shows up within minutes without an invalidation path.
async function boardingInfo(code) {
const key = `boarding:${code}`;
const started = Date.now();
const hit = await store.get(key).catch(() => null);
if (hit) return { ...JSON.parse(hit), cached: true, latencyMs: Date.now() - started };
const info = await lookup(code); // the slow path
await store.set(key, JSON.stringify(info), CACHE_TTL_SECONDS).catch(() => {}); // 180 s
return { ...info, cached: false, latencyMs: Date.now() - started };
}
05 — NOTHING IS DEPLOYED YET
At this point boarding-api has an image and no environment: nothing is
running anywhere. Onboarding creates the app, its repos and its pipeline; it does not
create a Deployment. Environments are separate objects, added next, and each tier is
configured in a different place.
You edit platform/envs/dev.yaml in your own repo (section 06). The
App Configuration tab can't see ground environments.
The tab edits gitops-<app>/<cluster>/<env>/values.yaml
through pull requests (section 07). It only lists environments your
cicd.yaml declares under deploy.upperEnvironments.
06 — GROUND
A ground environment is two things: a file that creates the namespace, and a pipeline stage that puts an image in it.
# envName, not env — env is a reserved key in the chart's container env-var list. # rollout: null must be explicit — omitting it makes the chart render its own # default rollout with an empty image (two InvalidImageName pods). envName: dev rollout: null
A dev-cluster-only ApplicationSet
(boarding-api-lower-envs) watches platform/envs/*.yaml in
your repo. Within a sync interval you get an empty namespace,
app-boarding-api-dev, with the baseline ServiceAccount and
NetworkPolicy and no workload.
Replace the scaffolded cicd.yaml with a flow that builds,
tests and deploys to dev:
apiVersion: platform/v1
kind: PipelineConfig
build:
agent: nodejs-20
unitTest:
enabled: true # runs ./test.sh, which the code you copied in provides
deploy:
lowerEnvironments: [dev]
pipelines:
ci:
trigger: { source: git, event: push, branch: main }
steps:
- stage: build
- stage: test
env: dev
- stage: deploy
env: dev
Changing cicd.yaml triggers section 03's resync: a new PR
regenerates .tekton/. Merge it. Separately, ArgoCD reads the new
cicd.yaml and provisions the deploy RBAC for
app-boarding-api-dev.
Push to main. The deploy stage
commits rollout.image.repository and rollout.image.tag
directly into platform/envs/dev.yaml on main — no
branch, no PR. git pull before your next push, or you'll be rebasing
over the bot's commit.
envName: dev
rollout:
image:
repository: ghcr.io/jfillman/boarding-api
tag: 0.1.0-a1b2c3d
Chart defaults cover the rest: two replicas, port 8080 named
http, no probes.
kubectl port-forward -n app-boarding-api-dev svc/boarding-api 8080:8080
Open http://localhost:8080. Look up AC123 twice:
the second answer is a cache hit, a few milliseconds instead of ~400. Press
Scan a boarding pass several times: the counter jumps around, because the two
pods each count on their own (counter: memory). Keep that in mind for
section 08. To change ground settings, edit platform/envs/dev.yaml
directly.
07 — FLIGHT
Four steps: declare the environment in cicd.yaml, create the
ApplicationEnvironment, configure it in Tower, then release an image
into it.
The release stage supplies the image to a flight env, and
Tower's App Configuration tab builds its environment list from
upperEnvironments:
deploy:
lowerEnvironments: [dev]
upperEnvironments:
- { name: staging, cluster: prod }
governance:
allowedCommitSigners:
- you@example.com # with no signers every release fails the
# commit-signature gate
pipelines:
ci:
trigger: { source: git, event: push, branch: main }
steps:
- stage: build
- stage: test
env: dev
- stage: deploy
env: dev
- stage: release
env: staging
Merge the .tekton/ PR this push opens. Glidepath's
examples/04-multi-env-promotion.yaml and
10-production-grade.yaml show the governance options.
| Field | Value | Notes |
|---|---|---|
| Name | boarding-api-prod-staging | The XR's own name. Convention is <app>-<cluster>-<env>, as with every existing env. |
| Namespace | app-boarding-api-cicd | The app's tenant namespace. The xr-requests AppProject only permits this one. |
| Owner | group:default/jfillman | Same as every existing env. |
| appName | boarding-api | Required. Labels the env and drives a deletion-protection Usage on the app. Not live-checked: a typo silently creates an env for an app that doesn't exist. |
| cluster | prod | Required. Live-checked against the cluster registry: must be type: upper and crossplaneReady. A dev cluster is rejected — the XR reports ClusterReady: False and creates nothing. |
| env | staging | Required. A DNS label, max 20 characters (^[a-z0-9]([-a-z0-9]*[a-z0-9])?$) — it becomes part of the namespace app-boarding-api-staging and a git path. |
| configMapGenerator | off | Opt-in to a Kustomize configMapGenerator source for this env's config files. Leave off unless you need it. |
The Crossplane Settings page can stay at its defaults. Submitting
opens a PR into gitops-cluster-dev-tenants
(tenants/boarding-api/xr-requests/boarding-api-prod-staging.yaml).
Merge it. Crossplane commits a bootstrap values.yaml to
gitops-boarding-api/prod/staging/ and an onboarding entry to
prod's own tenants repo, so the prod cluster's own ArgoCD picks
the env up — no cross-cluster credential is involved.
$ kubectl get applicationenvironment -n app-boarding-api-cicd NAME SYNCED READY COMPOSITION AGE boarding-api-prod-staging True True applicationenvironments.catalog.idp.io 2m
kubectl describe shows ClusterReady: True and
WorkloadDeployed: False. The bootstrap file is rollout: null:
a namespace, no workload.
Pick environment staging. The active environment is shown
prominently at the top; check it before saving. Turn on the Deployment switch —
an env with rollout: null deploys no Rollout, Service, HPA or PDB — and
fill in:
| Section | For boarding-api |
|---|---|
| Deployment | On |
| Scaling | Replicas 4 (a canary splits by pod count, so four gives readable 25% steps) |
| Resources | Requests 100m / 128Mi, limits 500m / 256Mi |
| Service | Port name http, containerPort 8080 |
| Health checks | Liveness and readiness: HTTP GET /healthz on 8080 |
| Canary steps | Leave the default, or build a weight / pause / analysis sequence |
Save opens a pull request into gitops-boarding-api —
never a direct commit, in any environment. The tab validates the change against the
chart's own values.schema.json first. Merge the PR. There is no image
field: the image comes from the release pipeline in the next step, so until it runs
the Rollout has no image to start.
Push to main. After build, test
and deploy to dev, the release stage opens a PR against
gitops-boarding-api setting rollout.image for
staging. Merge it as a signed commit — GitHub's merge button
produces an unsigned one (see Glidepath's docs/admin/commit-signing.md,
"Merge strategy matters"). The prod cluster's ArgoCD syncs it, and
WorkloadDeployed flips to True once a real Rollout exists.
08 — REDIS
Verified live: the component has been provisioned on both clusters and the gate board reads and writes through it. It needed one platform fix on the way: Bitnami removed its versioned images, so the Composition now pins the last chart and image that still pull.
Redis is an Attached-tier component: not a Bootstrap object (no Create
form) and not an Embedded setting. You add one by putting an entry in the
components: list of an environment's values. The chart renders it into a
Redis XR, whose Composition renders a provider-helm
Release of Bitnami's redis chart: standalone, one instance per
entry, password-protected, in your app's namespace.
| Cluster | provider-helm | Redis XRD |
|---|---|---|
| dev (arm64) | installed, healthy | installed |
| prod (amd64) | installed, healthy | installed |
Both tiers have what they need, both clusters run airframe v0.3.83 (which includes the Redis naming fix), and every image involved is multi-arch.
Name the component anything; we use cache. The Composition sets
the chart's fullnameOverride to that name, so the Service is
cache-master and the password Secret is cache. (Before v0.3.79
the name had to be redis; every cluster is past that now.)
components:
- type: redis
name: cache
spec: { size: small, persistence: false } # environmentRef is stamped for you
env:
- { name: REDIS_URL, value: "redis://cache-master:6379" }
# The password is in the Secret the component creates, named after it (here `cache`).
- name: REDIS_PASSWORD
valueFrom: { secretKeyRef: { name: cache, key: redis-password } }
Keep the rollout: block the deploy stage wrote; only add these
keys.
The component creates a Secret named after itself, and the container reads its
password from it directly with valueFrom.secretKeyRef. There is no step where you
copy the password into Infisical. This needs airframe v0.3.88 or later; on an older
pin the fallback is to copy it once into boarding-api's Infisical project as
redis-password and list it under secrets:.
Everything the component creates carries hangar.io/app=boarding-api, so one
selector finds the app and its cache together:
kubectl get all,pvc,secret,cm -n app-boarding-api-dev -l hangar.io/app=boarding-api
Once the pods restart, the header says cache: redis. Scan a
boarding pass repeatedly: the counter now climbs by exactly one each time, whichever pod
answers — and survives a pod restart if you set persistence: true.
Shared Redis: not supported yet
One Redis serving several apps is not a feature of this catalog: Redis has
no attach mode, and a standalone InfraService hosting the instance is
unproven. Skyport's plan uses InfraService for RabbitMQ and the OAuth
server once those components exist.
09 — CANARY
The gate board polls /api/whoami twice a second and draws a bar of
which version answered; the header colour comes from that version. Version 2 also
adds a "boarding group" line to lookups, so a canary changes behaviour as well as colour.
Unverified: no canary has been run with this app.
Section 07 done, image released. Port-forward to the staging Service and
open the board: one solid bar, v1.0.0: 100%.
Set "version": "2.0.0" in package.json, commit
and push to main. The pipeline builds, tests, deploys to dev and opens a
release PR for staging; merge it as a signed commit.
The Rollout doesn't replace v1; it steps. With the default canary steps and four replicas the bar goes from all-v1 to about a quarter v2, then half, then all. The chart has no traffic router, so the split is by pod count, not request weight. A step that pauses without a duration waits for you.
kubectl argo rollouts get rollout boarding-api -n app-boarding-api-staging --watch
kubectl argo rollouts promote boarding-api -n app-boarding-api-staging # if paused
Press Reset tally between steps to see the current mix rather than
a running average. Look up AC123: only v2 pods show the boarding-group
line, and answered by names the version you got.
Set the env's rollout.strategy: blueGreen and the chart
renders two Services, boarding-api (active) and
boarding-api-preview. Port-forward to the preview to see v2 at 100% while
active still shows v1; after promotion the bar flips in one step. To roll back, abort
the rollout or revert the release PR.
THE FINISHED SHAPE
Each environment carries its own Redis cache. Flight runs four replicas that a new release steps through. The dashed box is the canary, which hasn't been run yet.
CHEAT SHEET
After creating the app, and again after every cicd.yaml change. No merge,
no pipeline.
Ground needs platform/envs/dev.yaml plus a deploy stage.
Flight needs an ApplicationEnvironment, App Configuration and a
release stage.
Copy the demo code over the scaffold, but keep its Containerfile and
don't narrow build.platforms.
The pipeline writes the image into platform/envs/dev.yaml itself —
git pull before you push.
In platform/envs/*.yaml, env: collides with the chart's
reserved key, and rollout: null must be explicit.
App Configuration only lists envs from deploy.upperEnvironments;
ApplicationEnvironment rejects dev clusters; merge release PRs as signed
commits. Redis is a component: its Service is <name>-master and its
password is read with valueFrom. A canary splits by pod count, so use four
replicas.