Hangar Platform · Airframe · Developer Guide · Part 3

A shared message broker

Part 1 built a NodeJS service with a cache and part 2 a Spring Boot service with its own database, and ended on a problem: move a flight's gate and the gate board keeps the old one until a three-minute cache expires. This part fixes it. You provision skyport-broker (RabbitMQ) once, then attach flight-api as a publisher and boarding-api as a consumer, so the board is right the moment a flight changes.

What has and hasn't been verified

Walked live on the dev cluster (2026-09-25): creating the InfraService (which found a real gap, section 02), declaring the broker in its environment file, both apps attaching and getting their credentials, boarding-api connecting with its real credentials and waiting for the exchange, and the pipeline builds (which found the image-scan failure in section 04). Tested against real RabbitMQ: the component in both modes on both clusters, and every allowed and refused action in section 07. Unit tests: flight-api 21, boarding-api 18. Not walked: section 08, the flight environments on prod. End to end, live: a lookup was a cache hit on gate A1; after PUT …/gate to B2 the same lookup returned B2 at once with cached: false, and eventsReceived climbed, with both apps on their final images (flight-api 1.1.0 on Spring Boot 3.5.16).

Needs airframe v0.3.89 or later

The rabbitmq component arrives in v0.3.89 (section 08 needs v0.3.90). Do parts 1 and 2 first: this guide assumes boarding-api and flight-api are running in their dev environments and links back rather than repeating the mechanics.

THE PATH

You write two things; the platform does the rest

Provisioning the broker is one form and one file. Everything between them is Git and ArgoCD.

How the skyport-broker gets provisioned Six steps left to right. You create an InfraService in Tower, which commits a request to the tenants repo, which creates the gitops-infra repo. You then add platform/envs/dev.yaml to that repo, ArgoCD's per-app ApplicationSet picks it up, and a RabbitMQ XR brings the broker up. Steps one and four are yours; the rest are automatic. Tower create InfraService Tenants repo xr-requests/*.yaml Empty deploy repo gitops-infra-… You add a file platform/envs/dev.yaml ArgoCD lower-envs AppSet Broker running RabbitMQ XR Ready YOU YOU AUTOMATIC AUTOMATIC AUTOMATIC AUTOMATIC

01 — WHAT YOU'RE BUILDING

One broker, three namespaces, least privilege

This is the first time in the series that apps share something. The broker lives in its own namespace and isn't dedicated to any app: the point is that several talk through it. Each app gets its own broker user, and the broker itself, not convention, decides what that user may do.

flight-api publishes to the broker and boarding-api consumes from it Three namespaces. In the left one, flight-api publishes to the flights.events topic exchange in the vhost flights on the broker in the middle namespace. A queue named boarding.flight-events, bound with flight.#, delivers to boarding-api in the right namespace, which deletes the changed flight from its Redis cache. Each app namespace holds an attach component that creates its broker user and Secret. The broker lists the two app namespaces as allowed. APP-FLIGHT-API-DEV APP-SKYPORT-BROKER-DEV APP-BOARDING-API-DEV VHOST · FLIGHTS PUBLISH DELIVER flight.# DEL boarding:AC123 flight-api Spring Boot · publisher after the database commits flight-mq attach · publish flights.events flights.events topic exchange owned by flight-api boarding.flight-events queue · owned by boarding-api allowedNamespaces the trust boundary · 1 pod, RabbitMQ 4.2 boarding-api NodeJS · consumer retries until the exchange exists Redis cache cache-master · part 1 board-mq attach · queues boarding.* Message path Credentials and connection details

Shared, on purpose

A database or cache is dedicated to one app environment. A broker can't be: the publisher and every consumer must sit on the same one, so it lives in its own namespace and is owned by an InfraService.

One vhost per domain

A vhost is a namespace inside the broker, and messages don't cross between them. Apps that exchange messages must share one, so it's named for the domain (flights), not for an app.

Each app, its own user

Permissions are generated from a few names, never hand-written regexes. A leaked boarding-api credential can't forge a flight event or read anyone else's queue.

02 — CREATE THE BROKER

An InfraService: a deploy repo with no code

The broker is a component, like Redis and PostgreSQL, but a shared one, so it needs somewhere to live that isn't any app's environment. That's an InfraService.

Tower → Create → InfraService

FieldValue
Nameskyport-broker
devClusterdev
descriptionSkyport shared message broker (RabbitMQ)
visibilityprivate

dev exactly, as in part 1. Tower opens a PR against the tenants repo; merge it. Within a few minutes:

$ kubectl get infraservice skyport-broker -n app-skyport-broker-cicd
NAME             SYNCED   READY   COMPOSITION                    AGE
skyport-broker   True     True    infraservices.catalog.idp.io   2m

It created one repo, gitops-infra-skyport-broker, holding only a README.md and a stub cicd.yaml. There is no application to build, so there is no pipeline.

Found by walking it

If the request stays OutOfSync with InfraService is not permitted in project idp-onboarding, the dev cluster's onboarding project predates InfraService: add it to the namespaceResourceWhitelist in 02-argocd-apps/xr-requests/appproject.yaml (gitops-cluster-dev, or apron for a new cluster). InfraService had never been created on a live cluster before this walk, and this was the first thing that broke.

03 — DECLARE THE BROKER

One file in the new repo

An app's environment is a file, platform/envs/<env>.yaml, in the repo the app is built from. For an InfraService that repo is gitops-infra-skyport-broker itself. Create the file and push it to main.

gitops-infra-skyport-broker/platform/envs/dev.yaml ground
appType: infra
envName: dev
rollout: null            # no workload of our own: the broker is the component below

components:
  - type: rabbitmq
    name: skyport-broker # also the broker's Service name: skyport-broker.app-skyport-broker-dev.svc
    spec:
      mode: broker
      size: small
      instances: 1
      storageSize: 1Gi
      vhosts: [flights]
      # Only these namespaces may attach (get a user) and reach the AMQP port.
      allowedNamespaces: [app-flight-api-dev, app-boarding-api-dev]
FieldMeaning
mode: brokerThis entry runs the broker. (attach is what the apps use, below.)
sizesmall is 100m CPU and 512Mi to 1Gi of memory. medium and large scale that up.
instances1 is a single node. 3 is a replicated cluster (RabbitMQ needs an odd count).
vhostsCreated for you. One per domain.
allowedNamespacesThe trust boundary. A namespace not listed can neither create a user nor reach the broker's port. Adding an app to the broker is a one-line change here.

The dev cluster polls the repo, so the file is picked up within a few minutes, the namespace app-skyport-broker-dev is created and the component starts. The first start pulls the image and takes about two minutes:

$ kubectl get rabbitmqs.catalog.idp.io -n app-skyport-broker-dev
NAME             SYNCED   READY   COMPOSITION               AGE
skyport-broker   True     True    rabbitmq.catalog.idp.io   2m

$ kubectl get pods,svc,pvc -n app-skyport-broker-dev -l hangar.io/app=skyport-broker

Use the full name rabbitmqs.catalog.idp.io: plain rabbitmq is ambiguous with the operator's own API group. The broker's admin credentials are in the Secret skyport-broker-default-user; apps never need them.

04 — WIRE FLIGHT-API

The publisher

Two changes: the code, and the environment. The code is opt-in. With EVENTS_ENABLED unset it publishes nothing and needs no broker, so part 2's setup is unchanged.

Copy the code in

git clone https://github.com/jfillman/airframe.git /tmp/airframe   # skip if you still have it
cd flight-api
A=/tmp/airframe/examples/skyport/flight-api
cp $A/pom.xml .
cp $A/src/main/resources/application.properties src/main/resources/
cp $A/src/main/java/io/skyport/flight/{FlightMessage,FlightEventPublisher,NoopFlightEventPublisher,RabbitFlightEventPublisher,FlightService}.java src/main/java/io/skyport/flight/
cp $A/src/test/java/io/skyport/flight/{FlightServiceTest,RabbitFlightEventPublisherTest}.java src/test/java/io/skyport/flight/
./test.sh                # 21 passing

After the commit, never during

A change is written to flight_events in the same transaction as before; the message goes out only once that has committed. A change that rolls back is never announced.

A broker outage never fails a change

The publisher logs a warning and carries on. Losing one notification is better than rejecting a change that was already saved.

The message carries the new state

Each one has the flight's gate and delayMinutes after the change, so a consumer never has to call back to learn what is true now.

Routing key: flight.<number>.<type>

For example flight.AC123.gate_changed. Consumers bind flight.# for everything, or flight.AC123.* for one flight. flight-api declares the exchange itself; consumers only bind to it.

The pom also moves Spring Boot to 3.5.16

Not because of RabbitMQ. The pipeline's image scan fails the build on HIGH and CRITICAL findings, and Boot 3.3.4 (already end-of-life) now trips it on its own Tomcat, Spring and Jackson: 39 findings. Boot 3.5.16 leaves 8, and the pom pins Tomcat, the Postgres driver, the RabbitMQ client and Netty to patched versions for the rest. Part 2's flight-api would fail the same scan today. The <properties> in the pom say why each override exists; drop them as Boot catches up.

Attach it, and hand it the connection details

flight-api/platform/envs/dev.yaml ground
components:
  # ... flight-db stays as it is ...
  - type: rabbitmq
    name: flight-mq            # credentials land in Secret flight-mq-user-credentials
    spec:
      mode: attach
      brokerRef: { name: skyport-broker, namespace: app-skyport-broker-dev }
      vhost: flights
      publish: [flights.events]   # may declare and publish to this exchange, nothing else

env:
  # ... the DB_* entries stay as they are ...
  - { name: EVENTS_ENABLED, value: "true" }
  - { name: RABBITMQ_HOST,     valueFrom: { configMapKeyRef: { name: flight-mq-connection, key: host } } }
  - { name: RABBITMQ_VHOST,    valueFrom: { configMapKeyRef: { name: flight-mq-connection, key: vhost } } }
  - { name: RABBITMQ_USER,     valueFrom: { secretKeyRef:    { name: flight-mq-user-credentials, key: username } } }
  - { name: RABBITMQ_PASSWORD, valueFrom: { secretKeyRef:    { name: flight-mq-user-credentials, key: password } } }
An attach entry creates, in the app's own namespaceHolds
a broker usera generated name and password, in the Secret <name>-user-credentials (username, password)
a permissiongenerated from publish, consume and queuePrefix (section 07)
a ConfigMap <name>-connectionhost, port, vhost

Nothing is copied through Infisical: the app reads the Secret and the ConfigMap directly with valueFrom, exactly as it reads the database's credentials in part 2. Commit both, push, and let the pipeline run; git pull before your next push, because the deploy stage commits the image into platform/envs/dev.yaml.

05 — WIRE BOARDING-API

The consumer

The consumer is off unless RABBITMQ_HOST is set, so boarding-api still runs anywhere.

Copy the code in

cd boarding-api
A=/tmp/airframe/examples/skyport/boarding-api
cp $A/{app.js,index.js,events.js,store.js,package.json,package-lock.json} .
cp $A/test/{app.test.js,events.test.js} test/
npm ci && npm test        # 18 passing

Its own queue, then eviction

It creates boarding.flight-events, binds it to flights.events with flight.#, and on each message deletes boarding:<flight> from the Redis cache. The next lookup goes to flight-api and is correct.

Start order doesn't matter

It may only read the exchange, so it can't declare it. If flight-api hasn't started, the bind is refused (NOT_FOUND - no exchange 'flights.events') and it retries every five seconds until the exchange exists. Seen live.

Bad messages are ignored

A malformed message is dropped, not fatal; an eviction that fails is retried.

Visible in whoami

/api/whoami gains events (off, connecting, connected) and eventsReceived, so you can watch it work.

Attach it

boarding-api/platform/envs/dev.yaml ground
components:
  # ... the redis cache stays as it is ...
  - type: rabbitmq
    name: board-mq             # credentials land in Secret board-mq-user-credentials
    spec:
      mode: attach
      brokerRef: { name: skyport-broker, namespace: app-skyport-broker-dev }
      vhost: flights
      queuePrefix: boarding    # may create, bind and consume queues named boarding.*
      consume: [flights.events] # may bind them to this exchange, not publish to it

env:
  # ... REDIS_URL and FLIGHT_API_URL stay as they are ...
  - { name: RABBITMQ_HOST,     valueFrom: { configMapKeyRef: { name: board-mq-connection, key: host } } }
  - { name: RABBITMQ_VHOST,    valueFrom: { configMapKeyRef: { name: board-mq-connection, key: vhost } } }
  - { name: RABBITMQ_USER,     valueFrom: { secretKeyRef:    { name: board-mq-user-credentials, key: username } } }
  - { name: RABBITMQ_PASSWORD, valueFrom: { secretKeyRef:    { name: board-mq-user-credentials, key: password } } }

Commit, push, and let the pipeline redeploy.

06 — SEE IT WORK

The board is right at once

kubectl port-forward -n app-flight-api-dev svc/flight-api 8081:8080 &
kubectl port-forward -n app-boarding-api-dev svc/boarding-api 8080:8080 &

curl -s localhost:8080/api/whoami           # "events":"connected"
curl -s localhost:8080/api/boarding/AC123   # note the gate; a second call says "cached":true

curl -s -X PUT -H 'Content-Type: application/json' -d '{"gate":"B2"}' localhost:8081/api/flights/AC123/gate

curl -s localhost:8080/api/boarding/AC123   # gate is B2 at once, "cached":false
curl -s localhost:8080/api/whoami           # "eventsReceived" went up

In part 2 that last lookup showed the old gate for up to three minutes. Now the change reaches boarding-api as soon as it is committed. With two replicas the queue is shared, so one pod handles each event; the cache is in Redis, so evicting it once clears it for both. The simulator changes a flight about every 30 seconds too, so eventsReceived keeps climbing on its own.

07 — WHAT EACH APP CAN AND CAN'T DO

Permissions come from names, not regexes

Each attach entry is turned into a broker permission for that app's user. You never write the regex.

FieldGives the appFor
queuePrefix: boardingcreate, bind and consume queuesnames starting boarding.
publish: [flights.events]declare and publishthat exchange
consume: [flights.events]bind its queues to itthat exchange, read-only

Tested over real AMQP, with the credentials each app really uses

AttemptResult
flight-api declares flights.events and publishesallowed
boarding-api declares boarding.q, binds it, receives the messageallowed
boarding-api publishes to flights.eventsrefused
boarding-api declares a queue x.q, outside its prefixrefused
flight-api reads boarding-api's queuerefused

Who may attach is the broker owner's decision

That's allowedNamespaces (section 03). Anyone who can edit an app's environment file can request any permission on the vhost for that app, so it's the list of namespaces, not the permission text, that protects the broker. Fine for this demo; a shared production broker would also want an admission policy on what an app may ask for.

08 — FLIGHT

The same on the prod cluster

Not walked

This section is written from how the pieces work and from the component being verified on prod; it was not run end to end. The prod cluster was short of memory when this was written and the broker adds about 1Gi, so check the cluster's headroom first.

Create the environment

Tower → Create → ApplicationEnvironment: Name skyport-broker-prod-staging, Namespace app-skyport-broker-cicd, appName skyport-broker, cluster prod, env staging, and appType infra. Merge the PR, as in part 1, section 07. Needs airframe v0.3.90. appType tells the environment its deploy repo is gitops-infra-skyport-broker, not gitops-skyport-broker; without it the values file is written to a repo that doesn't exist and the environment reports Unready resources: app-usage. Both were found when someone created this environment.

Configure it in a pull request

In the PR that adds gitops-infra-skyport-broker/prod/staging/values.yaml, use the same components: block as section 03 with allowedNamespaces: [app-flight-api-staging, app-boarding-api-staging].

Attach the apps

In each app's staging values, add its attach component and the env entries from sections 04 and 05, with brokerRef pointing at app-skyport-broker-staging.

What was verified on prod (Calico, which enforces network policy): the RabbitMQ operators, cert-manager and the component are installed; broker and attach both reached Ready with the app baseline policy in place; a pod in an allowed namespace connected on 5672 and a pod in a namespace not on the list was blocked.

CHEAT SHEET

What will actually bite you

Broker and attach are one component

The broker is an InfraService with a rabbitmq component in mode: broker. Apps use the same type in mode: attach. In kubectl use rabbitmqs.catalog.idp.io.

allowedNamespaces first

Add an app's namespace there before it attaches; an unlisted namespace's user is refused.

Credentials come from two objects

Secret <name>-user-credentials (username, password) and ConfigMap <name>-connection (host, port, vhost). Read them with valueFrom; nothing goes through Infisical.

Permissions from names

queuePrefix, publish, consume. The broker's Service is <name>.<namespace>.svc:5672.

Consumers retry; producers own the exchange

A consumer can't declare the exchange and retries until the producer has, so start order is free. Publish after the commit, and never fail the change on a broker error.

The image scan gates the build

If it fails on Tomcat, Spring, Jackson or a driver you didn't touch, the base has aged: bump Spring Boot or override the vulnerable version in the pom.

RabbitMQ 4.2, not 4.1

Operator 2.23's startup probe needs an endpoint 4.1 doesn't have (the pod never went ready). The component pins the version, so you don't choose.

Deleting the environment deletes the data

The broker isn't backed up. Its PVC isn't labelled hangar.io/app, unlike its pods, Services and Secrets. The provenance gate on release PRs fails for unsigned commits; it did on boarding-api's own earlier release too.