Hangar

Docs · Airframe

Skyport — the demo system

Skyport is one small, believable airport system split across five services. Its job is to exercise every Airframe stack and every component in one place, so each feature has a real caller and a visible effect instead of a toy curl.

The code lives in examples/skyport/ in the airframe repo. Why here and not hangar: the demos are consumers of Airframe’s XRDs and versioned alongside them, while hangar is the docs umbrella. Each service directory is what you copy into the source repo Airframe scaffolds for you (see the quickstart, section 03).

Status (2026-10-16). All five services and both shared pieces were built and ran in the previous lab, on its dev cluster and its prod cluster’s staging, except skyport-auth, which ran on dev only. Every component exists in Airframe: Redis, PostgreSQL, RabbitMQ (broker + attach), MongoDB, and Dex (server + attach) as the OAuth server. Token enforcement (Phase 4’s second half) is written, on the apps’ dex-tokens branches, and Dex comes to the prod cluster with Apron’s prod profile; neither has run on a cluster yet. The AI-workload phases are not built. The lab is being rebuilt from scratch; nothing has been replayed on the new clusters yet. The order to stand it all up again is the Skyport runbook.

gate-api (NodeJS) and checkin-api (Python) are two small extra services, outside the original five, that took the release-file split end to end first (part 6, part 7).

The five services

Service Airframe stack Components What it does
boarding-api NodeJSApplication Redis · RabbitMQ (consumer) The passenger-facing gate board: flight/gate lookups, boarding-pass scans, and the canary visualizer. Caches lookups in Redis.
flight-api SpringBootApplication Postgres · RabbitMQ (producer) · OAuth (resource server) System of record for flights and gates. A scheduled simulator delays flights and changes gates, recording an event each time (published to the broker from Phase 2).
baggage-api PythonApplication RabbitMQ (consumer) · MongoDB Tracks each bag’s journey as a document. Consumes flight events to re-route bags when a gate changes.
skyport-auth InfraService dex (server) The OIDC provider (Dex, client_credentials only). Issues the tokens flight-api and baggage-api validate and boarding-api sends.
skyport-broker InfraService rabbitmq The shared message broker flight-api, baggage-api and boarding-api all attach to.

GoApplication is deliberately left out; the other four Bootstrap stacks (NodeJS, SpringBoot, Python, InfraService) are all used, and InfraService twice — it’s the stack for things several services share.

An InfraService is an empty GitOps deploy repo with no source, so the two shared pieces are components hosted there rather than code.

How they work together

                 browser
                    │
                    ▼
            ┌───────────────┐   token    ┌───────────────┐
            │  boarding-api │──────────▶ │ skyport-auth  │
            │  (NodeJS)     │            │ (OIDC)        │
            │  Redis cache  │            └───────┬───────┘
            └──┬─────────▲──┘                    │ JWKS
        GET    │         │ gate-changed          ▼
     /flights  │         │ (evict cache)  ┌───────────────┐   flight.*    ┌───────────────┐
               ▼         │                │  flight-api   │──────────────▶│skyport-broker │
                         └────────────────│  (Spring)     │               │ (RabbitMQ)    │
                                          │  Postgres     │               └──────┬────────┘
                                          └───────────────┘                      │ flight.*
                                                                                 ▼
                                                                         ┌───────────────┐
                                                                         │  baggage-api  │
                                                                         │  (Python)     │
                                                                         │  MongoDB      │
                                                                         └───────────────┘

A day in the life:

  1. Look up a flight. The gate board asks boarding-api. On a Redis miss it calls flight-api (with a bearer token from skyport-auth), which reads Postgres. The answer is cached for three minutes.
  2. Scan a boarding pass. boarding-api increments a per-flight counter in Redis. With two replicas this is only correct because Redis is shared — the gate board shows which cache mode it’s in, and in-memory mode visibly miscounts.
  3. Something changes. flight-api’s simulator moves a flight to another gate and publishes flight.gate-changed.
  4. Two consumers react. boarding-api evicts the cached entry, so the next lookup shows the new gate at once instead of after the TTL. baggage-api re-routes that flight’s bags and updates their documents in MongoDB.

The events are the point of the design: cache invalidation and bag re-routing are two unrelated consumers of one fact, which is exactly what a queue is for.

Which component does what

Component Used by Demonstrates
Redis boarding-api Cache-aside with TTL, shared counters across replicas
Postgres flight-api A relational system of record, schema migration on deploy, credentials read straight from the component’s Secret
RabbitMQ flight-api → boarding-api, baggage-api Fan-out events; a shared broker via InfraService
MongoDB baggage-api Document storage that doesn’t fit rows
Dex (OAuth server) flight-api, baggage-api (resource servers), boarding-api (caller) Service-to-service tokens; one issuer per cluster/env
SLO / RolloutWatch / SecretStore all five The components that already exist, on every service

The canary test bench

boarding-api is built to make progressive delivery visible. The gate board polls /api/whoami twice a second and draws a bar of which version answered:

  • The header colour is derived from the answering version, so a v1 → v2 flip is obvious without reading anything.
  • v2.0.0 adds a “boarding group” line to the lookup result, so a canary also changes behaviour, not just a number.
  • Each poll opens a fresh connection (Connection: close), so even through kubectl port-forward — which pins a connection to one pod — the tally reflects the pod mix instead of showing 100% of whichever pod you landed on.

The quickstart walks a real canary with it. Blue/green works the same way: the bar flips from 100% v1 to 100% v2 at promotion instead of ramping.

AI workloads: one per workload shape

Planned 2026-09-26. Skyport also runs six AI agents, one for each shape Autopilot supports, so every part of it has a real caller and a test that can fail. They read Skyport’s APIs, draft, store artifacts, ask a human and spawn narrower runs. None of them can apply a change.

Shape Agent What it does
Task flight-briefer An ops brief for one flight from the three APIs.
Session gate-copilot A chat in Tower for a gate agent; drafts announcements, never sends.
Service passenger-assistant Always-on flight-status chat for passengers. An ordinary application.
Scheduled delay-digest A daily delays and gate-changes report.
Event disruption-responder On flight.*.delayed it drafts rebooking notices; a trigger bridge (an Airframe app on the RabbitMQ attach) starts one run per unique message.
Team irregular-ops-team A planner with researcher, drafter and checker workers, each narrower than its parent.

Definitions and their tests exist (clearance/agents/skyport/, six Preflight cases); the runtime that would run them (Clearance, the AgentRun XR, a model proxy) is not built yet. The design, the safety demonstrations (prompt injection, redelivered events, an over-broad spawn, a sandbox the dev cluster cannot provide) and the build order are in hangar/docs/autopilot/skyport-ai-workloads.md. Diagram: hangar/docs/autopilot/diagrams/plan/07-skyport-ai-workloads.html.

Phases

Each phase ends with something you can run. “Previous lab” means it ran there; nothing has been replayed on the rebuilt clusters yet, and each part’s walkthrough is how that gets proven.

Phase Adds Must exist first Status
0 boarding-api (NodeJS) + Redis + canary UI — part 1 Redis component, provider-helm on the target cluster Built; ran on dev with Redis and on prod staging (previous lab). The canary itself has not been walked with this app.
1 flight-api (Spring) + Postgres; boarding-api calls it — part 2 postgresql component (built) Built; ran on dev and on prod staging with a two-instance database (previous lab). boarding-api’s staging call to flight-api was not exercised.
2 skyport-broker (RabbitMQ), flight events, cache eviction — part 3 rabbitmq component (built) Built; walked on dev; broker staging env existed on prod (previous lab). The prod half of part 3 was not walked end to end.
3 baggage-api (Python) + MongoDB, consuming flight events — part 4 mongodb component (built) Built; ran on dev and on prod staging with its MongoDB and broker attach (previous lab). One replica, by design (one queue).
4 skyport-auth and enforced JWTs — part 5 dex component (built; chosen over Keycloak); Dex on prod from Apron’s prod profile Built, not yet run. Dex server ran on dev (previous lab). Enforcement is on the apps’ dex-tokens branches: baggage-api and boarding-api tested; flight-api written but not yet compiled.
5 (optional) an nginx edge as a third InfraService nginx component; its scope is still undecided Planned
6 First agent: flight-briefer (task) Autopilot core: Clearance, AgentRun, a model proxy Planned; not started
7 gate-copilot (session) session channel and Tower’s Agent tab Planned
8 passenger-assistant (service) the chart’s agent: block; a Redis attach Planned
9 delay-digest (scheduled) trigger runner, artifact store Planned
10 disruption-responder (event) trigger bridge on the RabbitMQ attach Planned
11 irregular-ops-team (team) run.spawn, approvals Planned

Phases 6 to 11 are numbered in the order they will be built, and slot in after phases 3 and 4 because the agents read baggage-api. The two extra services, gate-api and checkin-api, are not phases: they are built and ran on dev and prod staging in the previous lab (part 6, part 7). Phase 5 (nginx) moves after them. The unified plan is hangar/docs/autopilot/roadmap.md.

RabbitMQ’s questions are settled (shared broker, provider-rabbitmq not used), and so are Mongo’s (dedicated per environment, like PostgreSQL) and the OAuth server’s (Dex, shared, client_credentials). What nginx covers is still open (hangar/docs/service-catalog-design.md).

Two architectures

The dev cluster (ground) is arm64 and the prod cluster (flight) is amd64. Everything in Skyport has to run on both:

  • Application images. The build stage builds linux/arm64 and linux/amd64 by default (build.platforms in cicd.yaml); leave it alone. The amd64 leg runs under QEMU on the arm64 build node unless the stage opts out, so it is slow, and a compiler crashing under emulation is a known failure mode. Where a build stage is architecture-neutral (Java bytecode, pure-JS dependencies, Go cross-compiled with GOARCH=$TARGETARCH), write it FROM --platform=$BUILDPLATFORM so it runs natively; the platform’s builder is BuildKit since 2026-10-17 (kaniko ignored this, so older notes say it does not work). Or compile in build.script, natively in the build agent, and keep the Containerfile a thin packaging step - flight-api does this. Under kaniko the old Java scaffold Containerfile, which compiled in a target-platform stage, took about 11 minutes for the emulated leg (about 17 for the whole image). boarding-api has no native dependencies. baggage-api is pure Python (pika), so it builds natively on either architecture. flight-api and baggage-api must avoid ones without both wheels or classifiers.
  • Python images and the scan gate (baggage-api). python:3.13-slim (Debian 13) failed the Trivy gate with 44 HIGH util-linux/acl CVEs that have no fix yet. python:3.13-alpine clears the OS findings, but the vendored copies of msgpack and setuptools inside pip and setuptools then fail it (GHSA-6v7p-g79w-8964, CVE-2025-47273); upgrading does not remove them, so the Containerfile runs pip uninstall -y setuptools pip after installing dependencies. The scaffold’s Containerfile should do the same (a scorecard item).
  • Replicas (baggage-api). Every replica declares the same queue and they compete for its messages, so with two replicas each applies half of the events. State is in MongoDB now, but this still caps it at one replica until the consumer changes.
  • First deploy. Configure an app before it has an image with rollout: null on chart versions before v0.3.91; from v0.3.91 the chart renders no workload until an image exists, so ordinary rollout config is safe. Since v0.3.140 rollout.enabled: false (not rollout: null) says an environment runs no service.
  • Component charts. Every image a wrapped upstream chart pulls has to be a multi-arch index. Checked for Redis (Bitnami redis:latest: amd64 and arm64); check each new component before it ships — this is the easiest thing to get wrong.
  • Crossplane Functions and providers. Any function or provider image this catalog publishes must be multi-arch, because Crossplane runs on both clusters. function-rollout-watcher already is. The platform toolbox image (platform-cicd-toolbox) is arm64-only, which is fine only because pipelines run on the dev cluster; a pipeline on the prod cluster would fail with exec format error.