Quick question. If someone told you right now that every API key in your production environment had been copied, how long would it take you to replace all of them? Not revoke. Replace, with the services still running. If your honest answer is "we would find out," you need a secret rotation playbook, and October 2026 is the month that should convince you.
In one week, credentials leaked from systems their owners did not run: a hijacked open source release, a hosting platform and an AI coding tool's hosted console. A secret rotation playbook means you know where every secret lives, you can swap any of them without a deploy, and you have timed the whole drill before you need it. The rest of this piece is how to get there.
What went wrong in October?
On 21 October, attackers used stolen developer credentials to push a poisoned LiteLLM 1.107.0 to PyPI, Docker Hub and GHCR, built from a lookalike repository. According to the LiteLLM incident report, the malicious wheels were on PyPI from 11:39 to 13:23 UTC. Anyone who ran it is told to rotate the host's cloud role, the provider keys in the proxy database, the database password and every environment variable the process could see.
The same day, Railway disclosed that an unauthorized party may have accessed encrypted customer environment variables, and decrypted some of them. Railway told all users to roll every variable. It also paused all deployments while it investigated. On Railway, a variable change is staged and only takes effect when the service deploys. Think about that for a second. The fix and the freeze pointed in opposite directions.
A few days earlier, OpenCode said an intruder reached a legacy system behind its Console on the night of 16 October, possibly exposing OpenAI, Anthropic and Google keys that about 1.4% of Console users had linked. Those users were emailed on 17 October. Some reported charges they did not make.
And for good measure, eMudhra revoked eight intermediate CA certificates after a researcher found on 17 October that their private keys could be recovered almost instantly. Different layer, same lesson. The secret you trust can break somewhere you never look.
Do you know where every secret lives?
You cannot rotate what you cannot find. Most teams have a vault and assume that is the inventory. It is not. Secrets also sit in CI variables, platform environment settings, a SaaS tool's "connect your provider" screen, a developer's shell history and the logs of whatever proxy touched them. LiteLLM's team believes its attacker pulled GitHub tokens out of logs.
Build a register. For each secret, record what it opens, who owns it, where copies exist and what breaks when it changes. A spreadsheet is fine. A current spreadsheet beats a product nobody keeps up to date.
Can you rotate a secret without a deploy?
This is where Railway customers got stuck, and plenty of self-hosted shops would get stuck the same way. If your apps read secrets once at boot from baked-in environment variables, every rotation is a release. Releases need pipelines, approvals and a platform that is working.
Have services fetch secrets at runtime from a store, cache them briefly and reload on change. Support two valid versions of a key at once, so you can add the new one, move traffic, then kill the old one. That overlap is what makes rotation boring. Boring is the goal.
Stop holding long-lived keys
The best secret to rotate is one that expires on its own. GitHub Actions can trade an OIDC token for short-lived AWS credentials. EKS workloads can get role credentials through pod identity instead of an access key sitting in a Kubernetes Secret. Databases can hand out per-session credentials through a broker. A leaked token that dies in an hour is a bad afternoon, not a breach notice.
You will not get everything there. Most AI provider keys are still static strings. Fine. Move what you can onto workload identity and shrink the pile you have to babysit.
Scope every key and cap the spend
One OpenAI key shared by six apps and two SaaS tools is a single point of failure with a credit card attached. Give each app and each tool its own key. Set per-key spend limits and usage alerts at the provider. When one leaks, you revoke one key, one service feels it, and the bill has a ceiling. OpenCode's advice to affected users came down to the same thing.
One more detail people miss. Deleting a key from a third-party console does not revoke it at the provider. Revoke at the source.
Rehearse it like a fire drill
Pick a quarter. Pick a class of secret, say every LLM provider key, and rotate all of them on a weekday afternoon with on-call watching. You will find the cron job nobody owns and the partner integration with a key pasted into a config file. Better to find them on a Tuesday you chose.
Write down what you actually did. That is your playbook. The one in the wiki that nobody has run is a wish list.
Then measure it. If you want one number for the leadership dashboard, use time-to-rotate: from "this credential is compromised" to "the old value no longer works and nothing is broken." Track it per class of secret. Hours is good. Days means you are counting on the attacker being slow.
That is the catch with incidents in other people's systems. You do not get to pick the timing. You only get to pick how ready you are.
