Hangar Platform · Airframe · Developer Guide

Airframe Quickstart

A hands-on walkthrough that ends with a running application: create a service, get its pipeline running, pull in the code, give it a ground environment to iterate in and a flight environment to run in, add a Redis cache, and finish with a real canary rollout you can watch. We build boarding-api, the gate board of Skyport: a small NodeJS service that caches gate lookups in Redis and has a page built to make canary and blue/green deployments visible.

What has and hasn't been verified

Verified: the app (11 tests, and it ran against a real Redis with two instances sharing one counter). Verified live: create → onboard → ground, and the Redis component: provisioned through the catalog on both clusters, the gate board reports cache: redis, and the password is read straight from the component's Secret. Walking this guide the first time found and fixed real problems (the Redis image, the devCluster name, the labels on the cache's resources), which is why the text names them. Not verified: the canary in section 09.

Two architectures

The dev cluster is arm64 and the prod cluster is amd64. Leave build.platforms unset in cicd.yaml so the image is built for both; narrow it and the other cluster gets exec format error.

01 — GROUND & FLIGHT

Two environments, built by different mechanisms

Every Airframe app can have two kinds of environment, with different owners and different review bars. Hangar calls them ground and flight.

Ground

On the ramp

What it is
A disposable environment on the dev cluster.
Declared in
platform/envs/<name>.yaml in your source repo
Gets its image from
Your pipeline's deploy stage, committed straight to main — no PR
Configured with
Editing platform/envs/<name>.yaml by hand
Cluster
Dev cluster only

Flight

Actually flying

What it is
A governed environment on a real upper cluster.
Declared in
An ApplicationEnvironment XR, rendering gitops-<app>/<cluster>/<env>/values.yaml
Gets its image from
Your pipeline's release stage, as a reviewed pull request
Configured with
Tower's App Configuration tab — each save is a PR
Cluster
Any cluster the registry marks type: upper

Both render through the same Helm chart, airframe-application. "Ground" and "flight" name the tier, not a field value — you'll still type real names like dev and staging below. Here is the whole path:

Path from new service to a canary rollout of boarding-api A flowchart of six steps: create the app, merge the pipeline onboarding pull request, pull in the code, add a ground environment, add a flight environment, then add a Redis cache and roll out a canary. New service Create the app NodeJSApplication · Tower Merge the .tekton PR Pipelines-as-Code Pull in the code examples/skyport/boarding-api Add a ground env platform/envs/dev.yaml Add a flight env ApplicationEnvironment Add a Redis cache components: [redis] Roll out a canary v1 → v2, watched live

02 — CREATE THE APP

boarding-api: the gate board

A gate screen looks up a flight, boarding-api returns the gate and boarding info, and caches the answer for three minutes so a busy screen doesn't hammer the reservations system. Its page also shows which version answered, which is what makes the canary in section 09 visible.

Tower → Create → NodeJSApplication

Every Bootstrap-tier stack — NodeJS, Spring Boot, Go, Python, InfraService — is a Backstage Scaffolder template generated from its XRD. Fill in:

FieldValue
Nameboarding-api
devClusterdev
descriptionSkyport gate board: Redis-cached flight lookups and a canary visualizer
nodeVersion20
packageManagernpm
port8080
visibilityprivate

Submitting opens a PR into gitops-cluster-dev-tenants at tenants/boarding-api/xr-requests/boarding-api.yaml. Merge it — the xr-requests ApplicationSet applies that file to the dev cluster as a NodeJSApplication XR.

Check the XR

$ kubectl get nodejsapplication boarding-api -n app-boarding-api-cicd
NAME           SYNCED   READY   COMPOSITION                         AGE
boarding-api   True     True    nodejsapplications.catalog.idp.io   3m

kubectl describe shows the two custom conditions this catalog adds: DevClusterReady and CicdOnboarded (Glidepath has picked the app up).

Crossplane creates the real boarding-api source repo and an empty gitops-boarding-api repo. The source repo starts with a hello-world index.js, a package.json, a Containerfile and a minimal cicd.yaml (build only, agent: nodejs-20, unit tests off). You have a repo — but no running pipeline, and nothing deployed.

Use dev exactly

Type dev for devCluster, even if your dev cluster goes by another name. The dev ApplicationSets hard-code cluster: dev when they render an environment's ExternalSecret, so it reads the store <app>-dev. An app created with any other value gets a store named <app>-<that value>, its ExternalSecret never syncs, and its pods sit in CreateContainerConfigError. And never remove dev from the cluster-registry: every existing app is pinned to it, and removing it turns their stores InvalidProviderConfig fleet-wide.

Starting over?

If a previous boarding-api was decommissioned, finish that first: delete its Infisical projects (boarding-api-dev, boarding-api-prod) and any boarding-api key in the prod cluster's secretstore-provisioner ConfigMap, or the new app tries to adopt a dead project. Move any old local boarding-api checkout out of the way too — section 04 clones into that name.

03 — PIPELINE ONBOARDING

Merge the .tekton pull request

Pipelines run through Pipelines-as-Code, which reads pipeline definitions from a .tekton/ folder in the source repo. You don't write those files — Glidepath does — but they have to get into your repo, and that takes one pull request that you merge.

Find the onboarding PR

Once the app is registered, Glidepath's onboarding hook notices your repo has no .tekton/onboarding-resync.yaml and opens a PR against boarding-api (and gitops-boarding-api) adding the generated .tekton/*.yaml files. Find it in GitHub or in Tower's Pull Requests tab.

Merge it

Until it's merged, pushing to main does nothing — PaC finds no pipeline to run. Once merged, a push runs the scaffolded build stage and publishes a multi-arch image to ghcr.io/<owner>/boarding-api.

These files are never hand-edited. Every later push that changes cicd.yaml opens a fresh PR regenerating .tekton/ if anything changed — merge those too. If nothing happens after you push, first check that the platform's GitHub App has access to the repo.

04 — PULL IN THE CODE

Replace the hello-world with the real app

The real boarding-api lives in the airframe repo under examples/skyport/boarding-api/. Copy it over the scaffold.

git clone https://github.com/jfillman/airframe.git /tmp/airframe
git clone https://github.com/jfillman/boarding-api.git && cd boarding-api   # fails if ./boarding-api exists

cp -R /tmp/airframe/examples/skyport/boarding-api/. .
git add -A && git commit -m "boarding-api: gate board with Redis-backed lookups"

This overwrites index.js and package.json and adds app.js, store.js, reservations.js, public/, test/ and test.sh. Keep the scaffolded Containerfile and, for now, cicd.yaml — you change it in section 06 and push then. Try it locally first:

npm install && npm test        # 6 passing
PORT=8080 node index.js        # open http://localhost:8080

With no REDIS_URL the header says cache: memory: the app falls back to an in-process store so it runs anywhere. With two replicas each pod keeps its own counts — the exact problem Redis fixes in section 08.

EndpointPurpose
GET /The gate board: flight lookup, boarding-pass scans, version tally
GET /api/boarding/:flightBoarding info for a flight like AC123; cached 3 minutes; reports cached and latencyMs
POST /api/boarding/:flight/scanIncrements that flight's boarded counter
GET /api/whoamiVersion, pod and cache mode — what the canary tally polls
GET /healthzLiveness / readiness

The cache-aside lookup at the heart of it, from app.js: check the cache, fall back to the slow path on a miss, set a short TTL so a gate change shows up within minutes without an invalidation path.

async function boardingInfo(code) {
  const key = `boarding:${code}`;
  const started = Date.now();
  const hit = await store.get(key).catch(() => null);
  if (hit) return { ...JSON.parse(hit), cached: true, latencyMs: Date.now() - started };
  const info = await lookup(code);                                  // the slow path
  await store.set(key, JSON.stringify(info), CACHE_TTL_SECONDS).catch(() => {});  // 180 s
  return { ...info, cached: false, latencyMs: Date.now() - started };
}

05 — NOTHING IS DEPLOYED YET

The app starts with no deployment

At this point boarding-api has an image and no environment: nothing is running anywhere. Onboarding creates the app, its repos and its pipeline; it does not create a Deployment. Environments are separate objects, added next, and each tier is configured in a different place.

Ground → edit the file yourself

You edit platform/envs/dev.yaml in your own repo (section 06). The App Configuration tab can't see ground environments.

Flight → Tower's App Configuration tab

The tab edits gitops-<app>/<cluster>/<env>/values.yaml through pull requests (section 07). It only lists environments your cicd.yaml declares under deploy.upperEnvironments.

06 — GROUND

A running environment on the dev cluster

A ground environment is two things: a file that creates the namespace, and a pipeline stage that puts an image in it.

Declare the environment

boarding-api/platform/envs/dev.yaml ground
# envName, not env — env is a reserved key in the chart's container env-var list.
# rollout: null must be explicit — omitting it makes the chart render its own
# default rollout with an empty image (two InvalidImageName pods).
envName: dev
rollout: null

A dev-cluster-only ApplicationSet (boarding-api-lower-envs) watches platform/envs/*.yaml in your repo. Within a sync interval you get an empty namespace, app-boarding-api-dev, with the baseline ServiceAccount and NetworkPolicy and no workload.

Let the pipeline deploy to it

Replace the scaffolded cicd.yaml with a flow that builds, tests and deploys to dev:

boarding-api/cicd.yaml ground
apiVersion: platform/v1
kind: PipelineConfig

build:
  agent: nodejs-20
  unitTest:
    enabled: true         # runs ./test.sh, which the code you copied in provides

deploy:
  lowerEnvironments: [dev]

pipelines:
  ci:
    trigger: { source: git, event: push, branch: main }
    steps:
      - stage: build
      - stage: test
        env: dev
      - stage: deploy
        env: dev

Commit both files, merge the .tekton PR

Changing cicd.yaml triggers section 03's resync: a new PR regenerates .tekton/. Merge it. Separately, ArgoCD reads the new cicd.yaml and provisions the deploy RBAC for app-boarding-api-dev.

Run it, then look at it

Push to main. The deploy stage commits rollout.image.repository and rollout.image.tag directly into platform/envs/dev.yaml on main — no branch, no PR. git pull before your next push, or you'll be rebasing over the bot's commit.

platform/envs/dev.yaml after the first deploy
envName: dev
rollout:
  image:
    repository: ghcr.io/jfillman/boarding-api
    tag: 0.1.0-a1b2c3d

Chart defaults cover the rest: two replicas, port 8080 named http, no probes.

kubectl port-forward -n app-boarding-api-dev svc/boarding-api 8080:8080

Open http://localhost:8080. Look up AC123 twice: the second answer is a cache hit, a few milliseconds instead of ~400. Press Scan a boarding pass several times: the counter jumps around, because the two pods each count on their own (counter: memory). Keep that in mind for section 08. To change ground settings, edit platform/envs/dev.yaml directly.

07 — FLIGHT

A governed environment on an upper cluster

Four steps: declare the environment in cicd.yaml, create the ApplicationEnvironment, configure it in Tower, then release an image into it.

Declare it in cicd.yaml

The release stage supplies the image to a flight env, and Tower's App Configuration tab builds its environment list from upperEnvironments:

boarding-api/cicd.yaml — additions flight
deploy:
  lowerEnvironments: [dev]
  upperEnvironments:
    - { name: staging, cluster: prod }

governance:
  allowedCommitSigners:
    - you@example.com   # with no signers every release fails the
                        # commit-signature gate

pipelines:
  ci:
    trigger: { source: git, event: push, branch: main }
    steps:
      - stage: build
      - stage: test
        env: dev
      - stage: deploy
        env: dev
      - stage: release
        env: staging

Merge the .tekton/ PR this push opens. Glidepath's examples/04-multi-env-promotion.yaml and 10-production-grade.yaml show the governance options.

Tower → Create → ApplicationEnvironment

FieldValueNotes
Nameboarding-api-prod-stagingThe XR's own name. Convention is <app>-<cluster>-<env>, as with every existing env.
Namespaceapp-boarding-api-cicdThe app's tenant namespace. The xr-requests AppProject only permits this one.
Ownergroup:default/jfillmanSame as every existing env.
appNameboarding-apiRequired. Labels the env and drives a deletion-protection Usage on the app. Not live-checked: a typo silently creates an env for an app that doesn't exist.
clusterprodRequired. Live-checked against the cluster registry: must be type: upper and crossplaneReady. A dev cluster is rejected — the XR reports ClusterReady: False and creates nothing.
envstagingRequired. A DNS label, max 20 characters (^[a-z0-9]([-a-z0-9]*[a-z0-9])?$) — it becomes part of the namespace app-boarding-api-staging and a git path.
configMapGeneratoroffOpt-in to a Kustomize configMapGenerator source for this env's config files. Leave off unless you need it.

The Crossplane Settings page can stay at its defaults. Submitting opens a PR into gitops-cluster-dev-tenants (tenants/boarding-api/xr-requests/boarding-api-prod-staging.yaml). Merge it. Crossplane commits a bootstrap values.yaml to gitops-boarding-api/prod/staging/ and an onboarding entry to prod's own tenants repo, so the prod cluster's own ArgoCD picks the env up — no cross-cluster credential is involved.

$ kubectl get applicationenvironment -n app-boarding-api-cicd
NAME                             SYNCED   READY   COMPOSITION                              AGE
boarding-api-prod-staging   True     True    applicationenvironments.catalog.idp.io   2m

kubectl describe shows ClusterReady: True and WorkloadDeployed: False. The bootstrap file is rollout: null: a namespace, no workload.

Configure it: Tower → boarding-api → App Configuration

Pick environment staging. The active environment is shown prominently at the top; check it before saving. Turn on the Deployment switch — an env with rollout: null deploys no Rollout, Service, HPA or PDB — and fill in:

SectionFor boarding-api
DeploymentOn
ScalingReplicas 4 (a canary splits by pod count, so four gives readable 25% steps)
ResourcesRequests 100m / 128Mi, limits 500m / 256Mi
ServicePort name http, containerPort 8080
Health checksLiveness and readiness: HTTP GET /healthz on 8080
Canary stepsLeave the default, or build a weight / pause / analysis sequence

Save opens a pull request into gitops-boarding-api — never a direct commit, in any environment. The tab validates the change against the chart's own values.schema.json first. Merge the PR. There is no image field: the image comes from the release pipeline in the next step, so until it runs the Rollout has no image to start.

Release an image

Push to main. After build, test and deploy to dev, the release stage opens a PR against gitops-boarding-api setting rollout.image for staging. Merge it as a signed commit — GitHub's merge button produces an unsigned one (see Glidepath's docs/admin/commit-signing.md, "Merge strategy matters"). The prod cluster's ArgoCD syncs it, and WorkloadDeployed flips to True once a real Rollout exists.

08 — REDIS

Adding a Redis cache

Verified live: the component has been provisioned on both clusters and the gate board reads and writes through it. It needed one platform fix on the way: Bitnami removed its versioned images, so the Composition now pins the last chart and image that still pull.

What kind of thing is Redis?

Redis is an Attached-tier component: not a Bootstrap object (no Create form) and not an Embedded setting. You add one by putting an entry in the components: list of an environment's values. The chart renders it into a Redis XR, whose Composition renders a provider-helm Release of Bitnami's redis chart: standalone, one instance per entry, password-protected, in your app's namespace.

Where it can work

Clusterprovider-helmRedis XRD
dev (arm64)installed, healthyinstalled
prod (amd64)installed, healthyinstalled

Both tiers have what they need, both clusters run airframe v0.3.83 (which includes the Redis naming fix), and every image involved is multi-arch.

Ground: add it to platform/envs/dev.yaml

Name the component anything; we use cache. The Composition sets the chart's fullnameOverride to that name, so the Service is cache-master and the password Secret is cache. (Before v0.3.79 the name had to be redis; every cluster is past that now.)

platform/envs/dev.yaml — additions ground
components:
  - type: redis
    name: cache
    spec: { size: small, persistence: false }   # environmentRef is stamped for you
env:
  - { name: REDIS_URL, value: "redis://cache-master:6379" }
  # The password is in the Secret the component creates, named after it (here `cache`).
  - name: REDIS_PASSWORD
    valueFrom: { secretKeyRef: { name: cache, key: redis-password } }

Keep the rollout: block the deploy stage wrote; only add these keys.

Nothing to copy

The component creates a Secret named after itself, and the container reads its password from it directly with valueFrom.secretKeyRef. There is no step where you copy the password into Infisical. This needs airframe v0.3.88 or later; on an older pin the fallback is to copy it once into boarding-api's Infisical project as redis-password and list it under secrets:.

Everything the component creates carries hangar.io/app=boarding-api, so one selector finds the app and its cache together:

kubectl get all,pvc,secret,cm -n app-boarding-api-dev -l hangar.io/app=boarding-api

See it

Once the pods restart, the header says cache: redis. Scan a boarding pass repeatedly: the counter now climbs by exactly one each time, whichever pod answers — and survives a pod restart if you set persistence: true.

Shared Redis: not supported yet

One Redis serving several apps is not a feature of this catalog: Redis has no attach mode, and a standalone InfraService hosting the instance is unproven. Skyport's plan uses InfraService for RabbitMQ and the OAuth server once those components exist.

09 — CANARY

Roll out a new version and watch it split

The gate board polls /api/whoami twice a second and draws a bar of which version answered; the header colour comes from that version. Version 2 also adds a "boarding group" line to lookups, so a canary changes behaviour as well as colour. Unverified: no canary has been run with this app.

Start with v1 in flight

Section 07 done, image released. Port-forward to the staging Service and open the board: one solid bar, v1.0.0: 100%.

Make v2

Set "version": "2.0.0" in package.json, commit and push to main. The pipeline builds, tests, deploys to dev and opens a release PR for staging; merge it as a signed commit.

Watch the canary

The Rollout doesn't replace v1; it steps. With the default canary steps and four replicas the bar goes from all-v1 to about a quarter v2, then half, then all. The chart has no traffic router, so the split is by pod count, not request weight. A step that pauses without a duration waits for you.

kubectl argo rollouts get rollout boarding-api -n app-boarding-api-staging --watch
kubectl argo rollouts promote boarding-api -n app-boarding-api-staging   # if paused

Press Reset tally between steps to see the current mix rather than a running average. Look up AC123: only v2 pods show the boarding-group line, and answered by names the version you got.

Blue/green, or roll back

Set the env's rollout.strategy: blueGreen and the chart renders two Services, boarding-api (active) and boarding-api-preview. Port-forward to the preview to see v2 at 100% while active still shows v1; after promotion the bar flips in one step. To roll back, abort the rollout or revert the release PR.

THE FINISHED SHAPE

What's running when you're done

Each environment carries its own Redis cache. Flight runs four replicas that a new release steps through. The dashed box is the canary, which hasn't been run yet.

The finished boarding-api topology Tower creates a ground environment on the dev cluster and a flight environment on the prod cluster. In each, boarding-api reads and writes a Redis cache component. The flight environment runs four replicas for a canary that has not yet been run. DEV CLUSTER PROD · UPPER CREATE GET / SET GET / SET Tower create app + envs boarding-api ground · 2 replicas app-boarding-api-dev Redis cache-master:6379 component · verified boarding-api flight · staging 4 replicas · canary Redis cache-master:6379 component · verified Provisioning (Tower) Runtime traffic Canary not yet run

CHEAT SHEET

What will actually bite you

Merge the .tekton PR

After creating the app, and again after every cicd.yaml change. No merge, no pipeline.

Nothing is deployed after onboarding

Ground needs platform/envs/dev.yaml plus a deploy stage. Flight needs an ApplicationEnvironment, App Configuration and a release stage.

Keep the scaffolded Containerfile

Copy the demo code over the scaffold, but keep its Containerfile and don't narrow build.platforms.

Ground deploys commit to main

The pipeline writes the image into platform/envs/dev.yaml itself — git pull before you push.

envName, never env; rollout: null

In platform/envs/*.yaml, env: collides with the chart's reserved key, and rollout: null must be explicit.

Flight rules

App Configuration only lists envs from deploy.upperEnvironments; ApplicationEnvironment rejects dev clusters; merge release PRs as signed commits. Redis is a component: its Service is <name>-master and its password is read with valueFrom. A canary splits by pod count, so use four replicas.