Every workflow in the org went red this morning and nothing in the repositories changed. Same commit that passed yesterday fails now. The step is the pnpm setup action, and the log is: It never reaches our install step,…
Everyone talks about slow CI, so I measured what large open source projects actually run. The last 100 completed runs from 24 well known repositories, split by workflow rather than lumped per repo. 24 repositories, 315…
Audited a publish pipeline after a bad release and found three separate things that could not fail. Different causes, one shape. 1. A secret with an invisible byte order mark. Written from PowerShell as , which prepends…
Roughly one in ten pipeline runs fails. Press retry and it goes green. Nobody has ever found a real bug behind one of these failures. The practical result is that a red build no longer means anything. People retry first…
Pushing to a remote fails with permission denied, publickey. I have generated a key, I have pasted the public half into my account on the hosting side, and I can see it listed there. Cloning over HTTPS works, so the…
Our build has been green for weeks. Nobody merged anything. This morning it fails, and it fails identically on re run and on branches that have not been touched in a month. Re running the last successful commit: the…
My pipeline needs a username and password for a registry, a token for a deployment target, and a key for signing. Right now some of these are variables in the project settings and one is, embarrassingly, in the pipeline…
I want a job to commit something back a regenerated file, a version bump, updated documentation. Inside the job, git is authenticated for reading but not for pushing, and my first attempt at adding a token worked and…
I want to run expensive steps only when the relevant part of the repository changed, build the frontend when the frontend changed, run the migration checks when migrations changed. Diffing against the previous commit…
I need conditional behaviour in a pipeline: run this job only on the main branch, run that one only for tagged releases, skip a slow step when only documentation changed, and run something different for merge requests.…
Team of nine, about 60 pipeline runs a day, suite is 22 minutes of which roughly 14 is integration tests hitting a local Postgres. Budget I can defend is around $300 a month. I can either pay for bigger hosted runners…
Our step reports a cache hit and restores in about 8 seconds, but the right after it still takes 2m40s and clearly unpacks everything again. The lockfile hasn't changed between the two runs I compared. Node 20, npm 10,…
Rolling update in Kubernetes, new replicas go Ready quickly, but for roughly 40 seconds after the rollout reports complete we still see responses from the old version, confirmed by a version header we stamp on every…
We passed an npm auth token as a for months. We've since moved to BuildKit secrets, but images built during that window are still in our registry and shows the value in plain text. How bad is this in practice, and is…
We run a full staging stack that mirrors prod's services and I can't remember the last time it caught anything real. Everything that breaks breaks in production, usually on data shapes we don't have in staging. Six…