Docs · Glidepath

ADR-0021 Phase 0 findings

2026-10-05. Spikes S1 to S5 from ADR-0021. S2 and S3 ran on dev in a throwaway namespace (spike-rollouts, label hangar.io/spike=adr-0021) with a request-catcher pod and a probe Rollout. dev’s argo-rollouts Application has no automated sync, so the config below was not reverted by Argo CD. S1 and S4 read prod; S4 ran one short-lived curl pod there.

Verdict

Phase 0 is complete. Every spike is resolved. The decision stands: facts from the Rollouts notifications engine, with the prune CronJob (option A). The inferred problem (S1) is now confirmed and attributed to the hooks.

S2 passes, with three design rules the ADR must adopt (A, B, C below). The Rollouts notifications engine delivers the facts we need. It does not guarantee delivery.

S2: the Rollouts notifications engine (Rollouts v1.9.1)

What arrived for each scenario (generation, phase, step, release-id):

Rules this adds to the ADR

A. Every trigger must require observedGeneration == string(generation). My first version did not, and it sent two Healthy facts carrying the new release-id and the old pod hash within a second of the patch, before the controller had observed the update. A reducer would have recorded the new release as Healthy before it started. With the guard added, no stale fact was sent in any later scenario.

B. The reducer must treat a fact as a release event only when the pod hash or release-id changed. Scale and restart produce facts at a new generation with an unchanged hash. They are ignored (or recorded as a scale), never as a new release. The pod hash is not an identity for “the same image”: use release-id and the image refs.

C. Delivery is not guaranteed, and a failure is silent. With the receiver scaled to zero, a real release (rel-5) went Progressing, Paused and Healthy, the engine logged “Sending notification” for each, logged no error, recorded all of them as notified, and never redelivered after the receiver came back (the catcher received nothing). This is the risk the ADR called out, and it is worse than assumed: not a delay, a loss. Mitigations, in order:

1. The stall alert (a record merged with no fact for N minutes) becomes mandatory, not an extra.
2. Heartbeat facts: a trigger that re-sends the current state once per time bucket makes the stream level-triggered, so a lost fact becomes a late one. The Rollouts controller re-reconciles every rollout every 15 minutes exactly (seen in the logs at :14:13, :29:13, :44:13, :59:13), so that is the heartbeat granularity. A poke (annotation write) triggers it immediately. See the open item below.
3. If a 15 minute recovery bound is not acceptable, fall back to the prod-side adapter named in the ADR, which can queue and retry.

Also seen: a stuck rollout (image will not pull) emits no failure fact until progressDeadlineSeconds (default 600 s) expires. The stall alert covers the gap.

S1: does a hook hold the Argo CD operation open?

Partly confirmed from data already on prod; the indefinite-pause case is not tested.

• Each release adds PreSync 3 to 7 s and PostSync 4 to 14 s of Job time to the operation, and the PostSync Job starts only after the Rollout is progressing and healthy (boarding: PreSync done 01:25:12, PostSync started 01:25:28, with a 30 s canary pause in between).
• The 14 to 16 minute operations are retry loops after failed syncs (each has a SyncFail Job in the middle), during which the operation stays Running. That is the same mechanism as the inferred problem: an in-flight operation delays the next sync.
• Not tested: a canary that pauses indefinitely or runs a long analysis. The staging canaries here pause 30 s. A direct test needs a throwaway Application and a git commit; deferred, because the retry-loop evidence already shows the operation staying open.

S3: facts per scenario

Covered in the S2 table. Not tested: kubectl argo rollouts undo under Argo CD selfHeal (needs an Argo-managed Rollout). The expectation in the ADR stands as unverified.

S4: reach the dev relay from the Rollouts controller’s namespace

A curl pod in argo-rollouts on prod reached http://192.168.1.78:30880/outcome/prod (the dev relay NodePort) with HTTP 405 for GET in 1.6 ms. There is no NetworkPolicy in that namespace. By IP it works; by dev.lab.internal it would not (pods do not resolve host /etc/hosts names). The webhook URL therefore needs the IP (which changes when the dev VM IP changes, as refresh-kiac-hosts.sh already handles for relayHostAliasIP) or a hostAliases entry on the single argo-rollouts Deployment. A real cluster uses a real hostname.

S5: Argo CD notifications for sync failure

prod runs two Argo CD instances: argocd (platform) and argocd-apps (tenant Applications, where the five release-tracked apps live). argocd-apps already has a running argocd-apps-notifications-controller and an argocd-notifications-cm holding only context. So the failure-only trigger needs config and a subscription, not a new controller. It uses the same notifications engine verified in S2. Not tested live: a sync-failure trigger firing (needs a PR to the cluster gitops repo, or a throwaway Application).

What to change in the ADR

1. Add rules A and B to decision 1 (trigger guard; reducer ignores unchanged-hash facts).
2. Replace “Delivery is at-least-once” in decision 6 with: delivery is at-most-once per fact and failure is silent; heartbeat facts and the stall alert are required, not optional.
3. Note two Argo CD instances on prod; the failure trigger goes on argocd-apps.
4. Keep status Proposed until the heartbeat redelivery is confirmed and the Argo CD OutOfSync question is checked.

Heartbeat result (resolved) and a new problem

• It works. With oncePer: string(time.Now().Unix() / 60) the engine sent a fact on the controller’s own 15 minute resync (23:29:13), with no change to the Rollout. A lost fact therefore becomes a fact at most about 15 minutes late.
• The cost is unbounded state. oncePer records one key per distinct value in the Rollout’s notified.notifications.argoproj.io annotation, and the engine never pruned it in this run (keys from generation 2 were still there at generation 12; 20 keys became 23 over the heartbeat tests). A heartbeat adds one key per resync, about 96 a day, at roughly 100 bytes each. Kubernetes caps total annotations at 256 KiB, so a tracked Rollout would hit the cap in about four weeks, after which writes to it fail.
• No oncePer does not help. A heartbeat with no oncePer sent one fact and then nothing: the engine fires once per false-to-true transition of the condition.
• Options: (a) keep the engine and add a small prod CronJob that strips the notified... annotation on a schedule. That bounds the size and also forces a resend of current state at the next reconcile, which is itself a heartbeat; it needs a Role to patch Rollouts. (b) Use the prod-side adapter named in the ADR: a small controller that watches Rollouts, keeps its own state, queues and retries. It is one more component but removes the silent-loss and annotation problems at the source. This is now a real choice for the owner, not a footnote.

Second round: an Argo-managed throwaway app (2026-10-06)

A throwaway Application on dev (spike-adr21, selfHeal and prune on) synced a probe Rollout from a scratch branch of this repo (spike/adr-0021, branch since deleted). It used the same notification config and request-catcher as before, plus a PostSync hook Job and Argo CD notifications.

S1 confirmed, and the hooks are the cause.

So with a PostSync hook a paused or long-analysis canary pins the sync operation and blocks a revert; without the hook it does not. This is the strongest argument for the change.

Other results

• Argo CD does not show OutOfSync for the notified... annotation. The Rollout stayed Synced throughout, including while the annotation held 32 keys.
• selfHeal reverts a manual spec change. A manual image patch (what kubectl argo rollouts undo does) was reverted by Argo within about 2 s. So undo does not work under selfHeal; abort (status only, not in the manifest) is the in-cluster emergency lever and the rollback PR is how git catches up. The facts for the revert carry the unchanged release-id, which is rule B’s drift case: observed image differs from the release’s image.
• The prune CronJob mechanism works (option A). Removing the annotation took it from 32 keys to 2, the engine immediately resent current state (a Paused fact), and the Rollout stayed Synced. The removal triggers the resend at once; it does not wait for the 15 minute resync.
• Argo CD sync-failure trigger works, with two corrections.
  1. phase stays Running through Argo’s automated retries (limit 5, backoff to about 3 minutes) and only becomes Failed after the fifth, so a phase in [Failed, Error] trigger fires about 5 minutes after the first failure, not on each attempt. The webhook body carried the full error message, revision and timestamps. (The hook-based SyncFail fires per attempt, which is earlier. A retry-count trigger would restore that; my one attempt did not fire and was not pursued. Open for phase 1.)
  2. oncePer: ...syncResult.revision made the trigger error (the log says FAILED, which here means an expression error, not “condition false”) because syncResult was missing for a failed sync. Use oncePer: ...finishedAt.
• Notification config written as inline JSON with \u0027 lost its quotes and errored the same way; write it from a file.

Open items

• Retry-count trigger for earlier failure facts: untested (see above).
• prod, not dev: the Rollouts notification config has not been installed on prod. That is phase 1 and goes through a PR to the cluster gitops repo.

Cleanup

Done 2026-10-06: dev spike-adr21 and spike-rollouts namespaces, the argo-rollouts-notification-configmap, the catcher-token key, and the four keys added to argocd-notifications-cm were removed; the remote branch adr-0021-spike was deleted. Left behind: a local worktree and branch (glidepath-wt-adr21spike, adr-0021-spike), which can be removed with git worktree remove and git branch -D. The curl pod on prod from S4 was deleted after its single run.