Glidepath already ran a TestWorkflow and the release stage already had a slot for a QA gate. The question was whether that is enough: give the business a standard test, give developers the implementation, and let the gate read the result. Reading the code and the live cluster, the answer was nearly yes. Three things were missing, and none of them is a test runner.
Since then. Phases 0 to 3 were built in Glidepath and verified live on boarding-api by October 19, 2026, with the real qa gate reading the check ride from the image's test provenance. Phase 4 shipped the same day in Tower v0.79 and v0.80: test evidence on the image and the Release Record, check-ride readiness and a Tests view under Pipelines. The reference for what is built is Glidepath's testing doc; this page is the proposal as written.
check-ride attestation on the image digest, made the same keyless way sast-scan already attests. The QA gate then verifies, with cosign and Rekor, instead of querying a database it has to trust.baggage-api, boarding-api, gate-api and sky-marshall all declare build → deploy test → deploy dev with no test step and no TestWorkflow file. The only TestWorkflows in the testkube namespace are two one-step curl smoke checks from demo apps that no longer exist, last executed 42 days ago. Tower's Release Record reads test results from Tekton Results and finds none, the qa gate is a declared stub, and Autoland is parked on "needs real test results". Every finding below was read in source, then checked against that cluster.
The wrapper is better than its reputation. run-testworkflow is careful about secrets, names, the CLI's log-streaming bug and Testkube's false-abort race, and it already emits a result and an event. What it lacks is a reason to exist: no app calls it, nothing it produces is addressed to the release, and the gate that should read it reads nothing.
All four Skyport apps: build → deploy test → deploy dev. No test: block, no glidepath/*.yaml workflow, no integration-test.sh. The live testkube namespace holds two orphan workflows from the retired demo apps.
Evidencecicd.yaml in baggage-api, boarding-api, gate-api, sky-marshall · kubectl get testworkflows -n testkube
run-testworkflow reports skipped when neither the workflow file nor the fallback script exists, and the stage succeeds. For an ops action that is right; for the test the release depends on it is a hole the SDLC review already flagged.
Evidencerun-testworkflow.yaml, fallback-script step
The Task receives image-ref, env and app-name and uses them only for logging and the fallback script. The TestWorkflow is applied as written: it cannot know which digest it is testing, which environment or URL to hit, or which commit it belongs to. Tags are never set, so the execution cannot be found by image later.
Evidencerun-testworkflow.yaml, execute-testworkflow step; Testkube execution.tags and config unused
qa-gate.yaml calls governance-stub. It has repo-url and revision threaded through for "reading the test stage's recorded outcome", but no recorded outcome exists that is keyed by image. Business rule GATE-01 lists it as a stub; GATE-03 means that on GitHub Free even a real gate would only advise.
Evidenceqa-gate.yaml · qa-check.yaml · pull-request-qa.yaml · business-rules.md §4, §16
useReleaseRecords.ts flattens every TaskResult of a test PipelineRun into name/value pairs and notes, in its own comment, that no source produces a pass/total tally. The comment is one release behind: run-testworkflow does write outcome and step-summary now. The record would show two strings, if any app produced them. Readiness (REL-11) has no test check; Autoland (REL-01) is parked on exactly this.
Evidencetower src/useReleaseRecords.ts buildTestResults · tower docs/release-records/index.md
The schema allows any number of test steps with a per-step testName, and the flow triggers key each step's CEL filter on the exact previous step (<flow>-<index>-<stage>-) after a live infinite-loop bug. What is missing is meaning: every test step is the same kind of thing, with the same cap and the same pass-or-skip rule.
Evidenceflow-triggers.yaml · cicd.schema.json pipelines.steps[].testName · _helpers.tpl validateFlows
Testkube 2.12.2 standalone: one shared namespace (ADR-0007), a Kyverno policy that pins a workflow to its own app's secret (ADR-0008), and a false-abort race the wrapper already works around by trusting the steps. The agent also emits webhooks and CDEvents, parses JUnit from artifacts, supports templates with typed config, execution tags, retries, services and matrix runs. None of that is used yet.
Evidencethe dev cluster testkube-api-server 2.12.2 · docs.testkube.io open-source, execution-tags, test-workflow-templates, test-workflows-artifacts
A test result lives in three places that each expire or stay on the dev cluster: the Tekton Results archive (30 days), the Testkube database, and a CDEvent the broker consumes once. The image that ships to prod carries SLSA provenance, a SAST attestation and an SBOM, and no record that it was ever tested.
Evidencetekton-results.md · image-signing.md · verify-sast-attestation.yaml
Yes, with one addition. A wrapper decides what goes in and what comes out; that is the contract. But a gate that reads a contract from a database has to trust the database, the namespace it lives in, and whoever can write to it. The release stage already solved that problem once for SAST: the build attests, the gate verifies the signature against the platform's own Fulcio and Rekor. The test stage should do the same. Then the contract has two halves, and the gate needs no new trust.
A fixed set of inputs every workflow can rely on (config and tags: app, image, digest, env, URL, revision, chain) and a fixed set of outputs the platform records (outcome, execution id, workflow revision, JUnit counts, report link). Three step roles: execute, test, check ride.
After a passed check ride, the wrapper signs a check-ride attestation onto the image digest, keyless, as the app's own pipeline-runner. The attestation travels with the image to every cluster and outlives every archive. The qa gate verifies it.
A shared check-rides repository: templates owned by SDETs and product, per-app workflows owned by the team and SDETs, synced by Argo CD into the testkube namespace, and admitted by the existing Kyverno policy extended to cover synced workflows.
Templates with typed config, step retries and timeouts, services and matrix runs, JUnit parsing, artifacts, schedules, webhooks and CDEvents, the execution record and a viewer. All in the free agent. Glidepath calls these; it does not rebuild them.
Four changes to Glidepath, one new repository, one new attestation type, and a Tower strip. No new stage, no new CRD, no new service.
Keep stage: test as the only DAG slot; the fixed superset DAG is a design rule worth keeping. Give the step a role that says what kind of thing it is, and pass the step's context into the workflow instead of applying the file blind.
Clear a cache, warm a pool, seed a fixture, run a migration dry-run. Pass or fail still chains the next stage. No attestation, no JUnit expected, no QA evidence. A missing workflow is a failure, since the step was written on purpose. required: false turns a failure into a warning for best-effort actions.
Integration, contract, smoke, load. The workflow lives in the app repo as today. Results are recorded and shown, and the next stage chains on them, but the qa gate does not read them. A missing workflow is a failure.
The workflow is <app>-check-ride from the shared repository, never from the app repo. Its passing run is attested onto the digest. Exactly one per flow, and validateFlows requires one after the last Ground deploy in any flow that ends in a release, once the catalog switches qa.required on.
| cicd.yaml | What it means |
|---|---|
| pipelines: ci: trigger: { source: git, event: push, branch: main } steps: - stage: build - stage: test env: dev testName: flush-cache role: execute required: false - stage: deploy env: dev - stage: test env: dev testName: integration # glidepath/integration.yaml timeout: 20m - stage: test env: dev role: check-ride # <app>-check-ride, shared repo - stage: release env: staging | One flow, three test steps with three meanings. The Every step, whatever its role, runs the workflow with the same config and tags: |
Unchanged: the env requirement, the chain on outcome == Succeeded, the secrets materialization, the name rewrite to <app>-<name>, and the false-abort workaround. Ground only for now: a check ride against a Flight environment needs a trigger on release healthy, which does not exist yet (roadmap phase 6).
A check ride is the flight test a pilot passes to be certified for a rating. Here it is the one workflow a release must pass, and the name separates it from the team's own tests. The repository is the shared ground: the business sets the standard in a template, the team implements it in a workflow that uses the template, and GitHub code owners make both signatures necessary.
| Path | What it holds | Owned by | How that holds |
|---|---|---|---|
templates/check-ride-http-service.yaml | The standard for an HTTP service: required config keys, a health step, a contract step that must produce JUnit, a time budget, tags, an artifacts step. Typed config with defaults. | SDETs and product owners | Both groups are code owners; a change needs one review from each. |
templates/check-ride-worker.yaml, …-function.yaml | One template per service class Airframe already knows (Container, Function, Worker), so the standard matches the shape of the app. | SDETs and product owners | Same rule. |
apps/<app>/check-ride.yaml | The app's implementation: use: [check-ride-http-service], its config (suite path, test image, timeouts within the template's budget), extra steps. Content comes from the app repo at the revision under test, via content.git. | The owning team and SDETs | The team writes it; an SDET reviews it. Scaffolded by onboarding as a PR, like the .tekton files. |
apps/<app>/waivers.yaml (later) | A time-boxed waiver of one named template step, with a reason and an expiry. The gate honours an unexpired waiver and says so in the PR comment and the record. | Product owner + SDET | A waiver is the only way to ship past a failing check ride without a bypass alert. |
CODEOWNERS, the ruleset | What makes the ownership real. On GitHub Free a private repo does not enforce it (GATE-03); a public repo or a Team plan does. The check-rides repo holds no secrets, so it can be public. | Platform admins | Generated by the same hack/apply-guardrail-ruleset.sh pattern. |
One Application in gitops-cluster-dev syncs templates/ and apps/ into the testkube namespace. The pipeline no longer applies a check-ride workflow; it runs the one that is there, and records the synced revision. A developer cannot run a hollow local copy and call it the check ride.
ADR-0008's policy keys the allowed secret on the caller's namespace, which works for pipeline-runner and not for Argo CD. Add a rule for the sync identity: a workflow it applies may reference only the secret of the app named in its hangar.io/app label, and the label must match the file's apps/<app>/ folder. Same policy file, one more rule, live-verified with kubectl --as as before.
The template declares the steps every service must pass and marks them as not optional. The app's workflow adds steps and fills config. Testkube merges them, so the team cannot remove a required step without changing the template, which needs the product owner.
Three records, written in one place (run-testworkflow) with the same fields. The attestation is the one the gate reads. The Task results are the ones Tower reads today. The CDEvent is the one the broker and any future subscriber reads. Keeping them identical is the whole point: one definition, three carriers.
| Field | Source | Task result | Attestation predicate | CDEvent content |
|---|---|---|---|---|
| outcome | Execution result.status, steps-trusted on abort | ✓ (exists) | ✓ | ✓ (exists) |
| workflow, role | Step | ✓ | ✓ | ✓ (test-name exists) |
| execution-id, started, finished | Execution record | ✓ | ✓ | ✓ |
| workflow-source, workflow-revision | Argo CD sync annotations on the TestWorkflow (check ride) or the app repo revision (own tests) | ✓ | ✓ | ✓ |
| template, template-revision | The resolved workflow's use | ✓ | ||
| image, digest, env, url, revision, chain | The config the step passed in; also set as execution tags | ✓ | ✓ (digest is the subject) | ✓ (image, env exist) |
| tests: total, passed, failed, skipped | JUnit reports Testkube parsed from artifacts (reports) | ✓ | ✓ | ✓ |
| steps[]: name, status, duration | result.steps | step-summary (exists) | ✓ | |
| report-url | Testkube execution URL, and the JUnit artifact path | ✓ | ✓ |
https://hangar.io/attestations/check-ride/v1An in-toto statement whose subject is the image digest. Signed keyless by the app's pipeline-runner through the in-cluster Fulcio, uploaded to Rekor, the same path and the same verification identity verify-sast-attestation already checks. Written only for role: check-ride and only on a pass; a failed check ride leaves no attestation and the gate fails for want of one.
Every execution carries app, image, digest, env, role and chain as Testkube execution tags. Tower can list an app's runs with one tagSelector query, and the gate's PR comment can link to the execution it verified.
outcome: skipped stays for test.enabled: false. A step whose workflow does not exist fails. A check-ride step whose synced workflow is missing fails with "no check ride for <app> in the check-rides repo", which is the onboarding message a new app needs.
Landing the real gate is the one-file change qa-gate.yaml was shaped for: replace the stub step with verify-and-check, flip the registry entry to real, add the check to the ruleset. The PR comment carries the counts and the link, and the Release Record freezes the same evidence.
| Knob · glidepath-catalog values | Default | Meaning |
|---|---|---|
qa.required | false until phase 3 | When true, validateFlows refuses a flow that ends in a release step without a check-ride step after its last Ground deploy, and the gate fails on a missing attestation. When false the gate reports and the step is optional. |
qa.workflowName | check-ride | The suffix of the standard workflow, <app>-<suffix>. |
qa.maxAge | 72h | How old the passing run may be, measured from the attestation to the PR. A digest tested on Monday may ship on Thursday; a month-old run is not evidence. |
qa.sourceRepo, qa.allowedRevisions | the check-rides repo · main | Which repository and revisions count. Pinning to main at sync time means a merged template change applies to the next run, not to runs already attested. |
qa.env | last Ground environment | Which environment's run counts. A later phase adds a Flight run before production. |
GitHub Free still applies (GATE-03): the gate reports and the bypass alert fires after a merge past it. That is true of every gate here and is the production roadmap's T1, not this design's problem to solve.
Beside the certificate-style verification strip: Check ride with passed/failed/skipped counts, duration, the template and its revision, the execution link, and the attestation's verification state read the same way the provenance panel reads SLSA (fresh from Rekor by log index). The team's own role: test runs appear below it as rows from the Task results, replacing today's flat list. Frozen into the committed record at seal.
"Check ride passed for the leading candidate" joins the readiness list, display only like the others. It reads the Task results via Tekton Results for the candidate's chain, so it works before any release PR exists.
A Tests sub-tab under Pipelines, from one backend route that lists Testkube executions by tagSelector app=<app> through the in-cluster API, read-only, with JUnit summaries and artifact links. The backend's read identity gains get on the Testkube service, nothing on secrets.
Autoland's eligibility list names "tests that report results". The check ride is that. Canary metrics on prod remain the other blocker.
Checked against docs.testkube.io and the 2.12.2 agent on the dev cluster. The third diagram in Hangar's testing set records how the first three seams were found.
| Capability | Used for | Availability |
|---|---|---|
TestWorkflowTemplate with typed config | The business standard; {{ config.url }} in steps | Free agent |
use: at the top level | Template sets pod, service account, required steps | Free agent |
Execution tags, tagSelector on the list API | Find a run by app, digest, env | Free agent |
JUnit parsing from artifacts (reports) | The counts in the contract | Free agent |
Step retry, timeout, optional, condition | Flaky-step budget inside the workflow, not in Tekton | Free agent |
services, parallel, matrix and sharding | Browsers, databases, big suites | Free agent |
| Schedules in the workflow, TestTriggers on events | Nightly soak without a Glidepath cron | Free agent |
| Webhooks and CDEvents on execution finished | Replace polling with an event into the broker (phase 6) | Free agent; needs the cdeventsTarget value |
executionNamespaces, per-tenant runs | Would remove the shared namespace (ADR-0007) | Pro only, confirmed in source |
config.*.sensitive per-run secrets | Would replace the Kyverno fence (ADR-0008) | Not in this CE build |
| Running context, actor identity on an execution | Who or what started a run | Pro only |
| Dashboard | A place to browse runs | Commercial; the OSS viewer uploads to view.testkube.io, so not for private results. Tower's Tests view instead. |
| Result status on a fast pod | The false-abort race | 2.12.2 bug, worked around by trusting steps; re-test on upgrade |
Phase 3 accepts a check ride run on the last Ground environment as evidence for the first Flight PR. That is the honest floor today: nothing can trigger a test on release healthy. The production PR should eventually require a run tagged env=staging. Decide whether that is a gate rule (fail) or a readiness row (display) when phase 6 lands.
A check ride tests the running service, so its steps should come from the shared repo and its target from config. If a team's suite lives in the app repo, the workflow pulls it with content.git at the revision under test. The template decides which is allowed; recommend: steps from the shared repo, suites from the app repo.
CODEOWNERS and the qa check both report without blocking on a private repo. The check-rides repo can be public (no secrets), which makes its ownership real now. The gate waits on T1 like every other gate.
Every passing check ride adds an attestation to the digest; a digest rebuilt and retested ten times carries ten. cosign handles many; the gate takes the newest that verifies. Prune with the image prune Tower already runs.
Synced check-ride workflows and team workflows sit in one namespace. The Kyverno rule keeps secrets apart; nothing keeps a team's workflow from naming itself other-app-check-ride except the name rewrite in the wrapper and the folder-to-label rule in the sync policy. Acceptable for one owner's apps, as ADR-0007 says; revisit with Pro or a second Testkube.
2.12.2 is from early September. The false-abort race, the reports field and CDEvents emission should all be re-checked on the version phase 0 pins. Upgrade first, then build on the fields.
Six phases in order, each with a test that says when it is done. Phase 0 is the cheapest and the most important: nothing in this design should be built before one real test has been seen to run and fail on the cluster. Phases 1 to 3 are Glidepath; phase 4 is Tower; phase 5 is rollout. Effort is a single-engineer estimate.
Give boarding-api a real test step and a real workflow (Playwright or pytest against the Ground URL, JUnit to artifacts). Upgrade Testkube to a current agent on the dev cluster. Make a missing workflow fail for role: test. Make the poll budget a step parameter.
Done when one green test PipelineRun on the dev cluster whose Testkube record shows tags and a parsed JUnit report, and one red run from a deliberately failing suite that stops the chain.
run-testworkflow.yaml · test.yaml · boarding-api cicd.yaml · gitops-cluster-dev testkube/application.yaml
Pass config and tags into every run. Define the result set and write it to Task results and the CDEvent. Add role, required, timeout to the schema and validateFlows. Emit the check-ride attestation on a passed check-ride run, keyless, like sast-scan. Rebuild the toolbox (schema is baked in). Document in cicd-yaml-reference.md and a new docs/admin/testing.md.
Done when cosign verify-attestation --type succeeds for the predicate on a real digest; Tower's existing flat list shows the new results without a Tower change.
cicd.schema.json · _helpers.tpl · run-testworkflow.yaml · stage-debrief.yaml · toolbox image
Create the repo with one HTTP-service template and boarding-api's workflow. CODEOWNERS with two groups. Argo CD Application into testkube. Extend the Kyverno policy for the sync identity and the folder-to-label rule. Onboarding scaffolds apps/<app>/check-ride.yaml as a PR. Add the repo to the rename map and the business rules register.
Done when a template change needs both reviews; a workflow referencing another app's secret is refused at admission with kubectl --as the Argo CD identity; testkube get tw shows the synced revision annotation the attestation records.
new repo · gitops-cluster-dev 30-policy + 50-platform-cicd · deliver-onboarding-files.yaml
Replace the stub step in qa-gate.yaml with verify-and-policy. Registry entry to real, ruleset regenerated, GATE-01 updated. PR comment with counts and the execution link. Flip qa.required once every app with a Flight environment has a check ride.
Done when a release PR for a digest with a passing check ride goes green with counts; the same PR for an untested digest goes red and names the missing step; /retest qa re-reads.
qa-gate.yaml · values.yaml releaseGuardrails · hack/apply-guardrail-ruleset.sh · business-rules.md
Primary test strip in the Release Record from the verified attestation, frozen at seal. Readiness row. Tests sub-tab from one backend route over the Testkube list API by tag. Autoland eligibility reads the new readiness row. Docs in tower docs/release-records.
Done when the record for a release shows the check ride's counts and a verified state; the Tests view lists boarding-api's runs with JUnit summaries in the local harness.
tower useReleaseRecords.ts · ReleaseRecordDetail.tsx · nextRelease.ts · backstage glidepathProvenance.ts
Check rides for baggage-api, gate-api and sky-marshall. Waivers file and the gate's waiver rule. Retire the integration-test.sh fallback once no app uses it.
Done when every Skyport app with a Flight environment has a green qa check on its next release PR; the business rules register lists TEST-01 to TEST-05 and GATE-12 as enforced or soft with evidence.
four app repos · check-rides repo · release-guardrails.md
A Flight check ride on release healthy (a new trigger kind from the relay's facts), required for the production PR. Testkube CDEvents into the broker to replace polling. Nightly soak from a Testkube schedule, attested with role: soak for the record only. Matrix and sharded check rides from the template.
Done when the production release PR shows a staging check ride; the test stage no longer polls.
flow-triggers.yaml · glidepath-relay · testkube cdeventsTarget
Proposed IDs for docs/policy/business-rules.md, to be written with their strength and evidence as each phase lands. Nothing below is a rule yet.
| ID | Rule | Strength when built | Held by |
|---|---|---|---|
| TEST-01 | A test step runs the named TestWorkflow with the platform's config and tags (app, image, digest, env, url, revision, chain). A workflow never hardcodes its target. | Enforced | run-testworkflow |
| TEST-02 | A test or execute step whose workflow does not exist fails. Only test.enabled: false skips. | Enforced | run-testworkflow, validateFlows |
| TEST-03 | The check ride for an app is apps/<app>/check-ride.yaml in the check-rides repo, using a template from templates/. A change to a template needs an SDET and a product owner; a change to an app's workflow needs the team and an SDET. | Convention on GitHub Free; Enforced with code owners | CODEOWNERS, ruleset |
| TEST-04 | A synced workflow may reference only the secret of the app in its label, and the label must match its folder. | Enforced | Kyverno testkube-secret-usage |
| TEST-05 | A passed check ride is attested onto the image digest by the app's pipeline identity. A failed one leaves nothing. | Enforced | run-testworkflow, Fulcio, Rekor |
| GATE-12 | A release PR passes qa only if a verified check-ride attestation for the promoted digest meets the policy: the app's workflow, from the check-rides repo at an allowed revision, on the required environment, younger than qa.maxAge, with no failed tests, or an unexpired waiver that is named in the comment and the record. | Soft on GitHub Free (GATE-03); Enforced with the ruleset | qa-gate |
The Glidepath catalog (run-testworkflow, test, qa-gate, qa-check, resolve-promoted-image, verify-sast-attestation, the flow triggers and validateFlows), the cicd.yaml schema and reference, ADR-0007 and ADR-0008, the release-guardrails and release-state docs, Tower's useReleaseRecords.ts and release-records doc, the business rules register on its branch, the SDLC review and production roadmap, and the four Skyport apps' cicd.yaml. Live on the dev cluster: the Testkube deployment version, the installed CRDs, the two remaining TestWorkflows and their executions. Testkube capabilities come from docs.testkube.io for the current agent; the 2.12.2 agent on the cluster predates some of them and phase 0 upgrades it before anything depends on a field.
Nothing was changed in any repository or cluster.