Going Off The Rails

Between a pile of critical CVEs for PHP and Apache Traffic Server, and discovering that eMudhra had issued SSL certificates like 4, the internet had a wild time last week. Unfortunately, my company had a similar experience. I’m here (with permission from my boss and Oasis) to share the story.

For background, I’ve been working as the lead software engineer at a construction company called Oasis Management Inc. as it’s been digitizing its infrastructure. We’ve created an entire enterprise management platform, complete with an Android mobile app, all deployed through Railway. Things have been going so well that we’ve recently been picking up technical consulting work from other companies, so top brass at Oasis decided that the tech team should become its own consulting firm called Atlas, with Oasis as our primary client.

Here’s where the story really starts. Around 9pm on October 21st, 2026, an email from Railway hit my inbox with the subject line Urgent: Railway Security Disclosure - Rotate Credentials. The body was exactly as bad as you’d expect:

Earlier this morning, Railway detected anomalous activity involving systems that store customer environment variables. After further investigation, our investigation to date indicates that an unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents. Environment variables may contain secrets such as API keys or database and registry credentials.

…

What you should do

• Immediately revoke and reissue all environment variables, including any API keys, stored in Railway.
• If you have any credentials on an external registry, . [sic]
• Review logs for those services for unusual activity.

Until builds are restored, you will not be able to roll out new versions. We apologize for this inconvenience. Follow this incident to be notified when builds resume and for further updates.

This was a pretty huge email to receive after working hours. As a small consulting firm, we didn’t have a dedicated DevOps team; my boss (who is almost strictly on the business end) was busy, and I had gone to sleep early.

But, as luck would have it, I woke up a little after 1am, read the email, and realized the extent of our exposure. Because Railway had disabled deploys, the only options were:

1. Leave our website and database up with sensitive environment variables potentially compromised, or
2. Go offline indefinitely while Railway patched their systems.

Although Railway’s disclosure was vague as to whether our project had been affected, the responsible thing to do is always assume that our keys and secrets had been compromised. In that case, an attacker could access our database and use our decryption keys that Railway may have leaked to access encrypted business data. That alone was unacceptable — not to mention that the API keys we used had also been exposed, allowing attackers to use our AI tokens, mobile app notifications, and potentially manipulate our finances through Stripe. After several failed attempts to call my boss, I made the decision to take down our website and public database access just after 2am on October 22nd.

The remainder of the night was spent regenerating and rotating all of our environment secrets (securing our database and its contents in the process) and every API key I had access to while agents worked in parallel to analyze our network logs to see if anyone had exploited our database.

Having replaced as many keys as I could, I was relieved to discover that none of our logs were out of order: thankfully it was a slow week for Oasis office staff, who had barely accessed the database at all since Railway’s breach, and we could manually confirm the activity in the logs. The rest of the logs were similarly clear. With nothing left to do, I wrote up a massive text to my boss summarizing Railway’s disclosure and recommending next steps (rotating API keys only he could access) and got some sleep.

Oasis’s construction workers wake up pretty early, and immediately noticed that our mobile app wasn’t working. My boss was woken up by nearly 40 phone calls from panicked employees worried that they wouldn’t be able to clock in; he quickly assured them that we’d make sure our records were updated manually, and then finished out the API key rotations, closing our last vulnerabilities.

When I finally got his phone call, I was dreading that my boss would be angry at my decision to take the website down. He’d always been exceptionally level-headed, but taking the entire website down until who-knows-when went far beyond anything either of us had done before, and I’d done it without being able to get a clear confirmation.

Instead, he appreciated the multiple attempted 2am phone calls, was impressed by my “snap decision” in protecting our clients, and told me that Oasis’s employees had all been reassured. From there, the only thing to do was redeploy.

By the end, it took over 40 hours for Railway to bring deploys back online. Thankfully, by the morning of October 23rd, every Oasis worker was able to log into our mobile app and website as if nothing had changed.

So, what have we learned?

• We need systems in place to rapidly deactivate and rotate keys. More than one qualified person should have authority to rotate our keys, according to some agreed-upon rules and runbook.
• We need to agree in advance who can take production down and under what circumstances.
• We need a way to notify Oasis employees if either our website or mobile app (or both) isn’t available.

After the postmortem, we also determined that it would be good to move from Railway sooner rather than later. We had initially used them as our continuous deployment system because it was extremely convenient and user friendly for the non-technical upper management, and much faster to set up compared to a VPS and our own deployment infrastructure.

But since I got hired, I’ve containerized the entire application in Docker and found myself enjoying platform work, and my boss has been needled more and more by Railway’s monthly bills as our deployment velocity increased. Moving from Railway had always been something we wanted to investigate, but a switch is looking increasingly likely: it would cut our bill, our infrastructure is less likely to be targeted than Railway’s, and we’d prefer more control over our deployments.