Here is a question worth asking before your next incident, not during it. If your platform provider tells you to rotate every secret right now, and the same provider has switched off builds, how do you get the new key into your running service?
That was the position many customers were in after the Railway breach disclosed on 21 October 2026. An attacker reached the systems that hold customer environment variables and decrypted some of them. Railway told customers to rotate everything, while deployments were paused. On Railway, a changed variable reaches a service through a deploy. So you could rotate and break your app, or stay up on keys that might already be in someone else's hands.
The lesson is a design one. If secret rotation depends on your deploy pipeline, you cannot rotate when the pipeline is part of the incident.
What happened at Railway?
According to Railway's preliminary report, monitoring paged on-call at 13:58 UTC on 20 October for elevated latency. Engineers found an attempted privilege escalation in build infrastructure and halted build pipelines. Overnight they found more access routes and halted builds again.
By the morning of 21 October the attacker had pulled encrypted environment variables for a subset of workspaces. Then the attacker used CI credentials to decrypt artifacts holding more API access, ran unauthorized queries against the encrypted variables and decrypted some of them. Railway asked upstream vendors to revoke keys at 09:25 UTC and published its disclosure that evening.
Railway says at most 1.85% of workspaces had variables potentially exposed, which it put at about 100,000 workspaces. The targets were AI provider and platform keys, including GitHub, GitLab, Anthropic, OpenAI, OpenRouter, Stripe and Google. Railway believes the goal was to use AI keys for model distillation.
It was not only API keys. Researcher Taylor Monahan told The Hacker News that attackers drained crypto wallets whose private keys were stored on Railway, with the first known theft at least two hours before the public disclosure.
Running services stayed online. Builds came back for Pro and Enterprise customers early on 22 October, and for everyone else later that day.
Why could customers not just rotate?
Railway's documentation says adding, changing or removing a variable creates staged changes that you review and deploy. That is a reasonable design on a normal Tuesday. Config changes go through the same path as code, you get an audit trail, and nothing changes underneath a running process.
It is a terrible design on the day the build system is the crime scene.
Customers had three options, none good. Revoke the old key at the provider and watch the service fail, which one customer on Hacker News described doing. Leave it in place and hope. Or move the workload somewhere else in a hurry. Meanwhile, Railway itself had notified providers about every key it stored, so some keys were revoked upstream whether the customer was ready or not.
That last part is worth sitting with. The person who rotated your secret might not be you.
Should rotation be separate from deployment?
Yes. Rotation is an incident response tool, and incident tools have to work when other things are broken. Your deploy pipeline is one of the most likely things to be broken, or untrusted, during a serious incident. It holds credentials, it runs code, and attackers go after it for exactly that reason.
A few patterns get you out of the trap:
Fetch secrets at runtime. Have the service read secrets from a store such as AWS Secrets Manager, Vault or your cloud's equivalent, with a short cache and a reload signal. Changing the value then needs no build.
Support two live keys. If your code accepts a primary and a secondary credential, you can add the new key, switch over and revoke the old one without downtime.
Prefer short-lived credentials. Workload identity, OIDC federation and tokens that expire in an hour leave much less to steal from an environment variable store.
Keep a break-glass path. Know how to push a new secret to a running service by hand, and write it down where you can find it at 3am.
None of this is new. Most teams skip it because environment variables on a PaaS are easy, and easy wins every planning meeting until the bad week.
What about the secrets you cannot rotate?
Some things should never sit in a platform's variable store. A crypto wallet private key is the obvious one. You cannot rotate a wallet key. You can only move the funds, and the attackers got there first.
The same thinking applies to long-lived signing keys and root database passwords. If losing it means a permanent loss rather than an afternoon of rotation, it belongs in a system built for that job, with its own access controls, not in an env var next to your feature flags.
What to do this week
Railway recommends revoking and reapplying sensitive variables stored before 21 October, removing public network access to databases, and reviewing GitHub and registry audit logs. Do that if you run there.
Then run the drill everyone else should run. Pick one production service, pretend your deploy pipeline is down, and try to rotate its most valuable secret. Time it. If the honest answer is "we would wait for builds to come back", you have found your next project.
Railway says it is moving to single-tenant build infrastructure and adding a way to mass-revoke app permissions. Good. Your rotation plan should not depend on any provider getting that right.
