API · 03 of 03 · Observability

One call, seen three ways

The audit line, the metric and the trace carry the same operation name and the same trace id, so one slow click in Tower can be followed to the source that made it slow.

One call, seen three ways A call starts as a span in Tower on the prod cluster, passes the API server proxy with its traceparent unchanged, becomes a span in glidepath-api named for the operation, and has child spans for what the API reads: Tekton Results, the broker, GitHub. It leaves three signals: an audit line on stdout with the trace id, Prometheus metrics on port 9090 (requests, duration, streams), and a trace in Tempo whose id comes back in X-Glidepath-Trace-Id. Sloth turns the metrics into two SLOs, 99.5% availability and 95% of calls within 2.5 seconds, with burn-rate alerts. The Glidepath API dashboard in Grafana shows Tower's latency beside the API's own. ONE CALL, ACROSS TWO CLUSTERSWHAT THE CALL LEAVESWHAT PEOPLE LOOK ATTowerspan, on the prod clusterAPI server proxytraceparent passesglidepath-apispan named for the opChild spansResults · broker · GitHubAudit lineJSON on stdout, with traceIdMetrics on :9090requests · duration · streamsTrace in TempoX-Glidepath-Trace-IdSloth SLOs and alerts99.5% available · 95% within 2.5 sGrafana: Glidepath APITower's latency beside the API's ownThe gap between Tower's number and the API's is the network between clusters.LEGENDThe callerThe API, and its dashboardSignalsHTTP, traced

Measured where it is felt

Tower's own span metrics, read from the prod cluster through Thanos, sit next to the API's numbers on one dashboard.

An SLO that stays honest

The latency SLO burned at about 1.8 on the dev cluster from day one, from slow TaskRun lists. The objective was left alone.

Errors name their trace

Tower ends an error message with (trace <id>), so a screenshot is enough to find the call.

Hangar · Glidepath · API