Builds felt slow, and I didn't want to guess why. So I pulled every PipelineRun from Tekton Results and measured where the time went. The answer surprised me: outside the build itself, most of a pipeline's wall-clock was pods starting up to do almost nothing.
The build stage is the only pipeline where most of the wall-clock is real work. In most of the others, more than three quarters of the time went to scheduling pods and starting containers. Eight of the nine recommendations shipped the next day without dropping a single result, event or gate. The ninth, building amd64 images under emulation, is deferred, and it's still the biggest remaining cost.
Across all 7,414 TaskRuns, the gap between a TaskRun starting and its first step running was 14 seconds at the median, 40 at p90 and 105 at p99. That tracked node load, not pipeline design: the daily p90 swung from 15 seconds to almost six minutes.
Each one had to keep every existing feature, result name, event and gate.
The amd64 leg is cross-built under QEMU and is the slowest step on the platform: p90 664 seconds, worst case 46 minutes. Result: Backstage's build is still 25 minutes, 11 of them here.
npm and Maven caches were keyed on the whole lockfile, so any dependency bump started from empty. Result: One shared cache directory per tool, also mounted for unit tests.
Four to six pods before the work and five to six after, each paying a 14-second start-up tax to do under two seconds of work. Result: 13 pods to 4; a function build from 258-409s to 129s.
Every scan pulled a 120 MiB database, and each image was analysed three times. Result: A shared Trivy server: the scan step takes 4 seconds.
Four gates full-cloned a repo to read one file, and one failed 48 percent of the time for a reason unrelated to security. Result: 135 of 284 failures were Release Record PRs, now excluded. A stub check retired (ADR-0025).
A StatefulSet and a placeholder pod per run, buying nothing on a single node. Result: 10 to 25 seconds and one StatefulSet less per run.
Eight tasks cloned full history to push one commit. Result: Target resolution reads the file over the API: 1 second.
A pushed deploy sat unseen for up to two minutes waiting for the next poll. Result: A hard refresh, scoped by an admission policy: picked up in 4 seconds.
Rules downloaded and metrics posted on every scan. Result: Metrics off. The fetch turned out to be under a second, so vendoring was dropped.
The scheduler and controller manager restarted 120 times in 16 days on lease timeouts. Each restart stalls scheduling for the whole cluster.
No requests on any Tekton container meant the lowest CPU share under contention. A cluster default now gives every step a small request.
Builds skipped when the pushed head was a merge commit, a grant that could have rewritten an Application, and a fallback to defaults on an unknown revision.