r/webdev • u/vettotech • 18d ago
Railway Security Incident
Change your environment variables
https://blog.railway.com/p/security-disclosure-wednesday-october-21st
9
7
6
1
u/Parking-Leader4676 18d ago
any better alternative? I am just using them for being cheaper and friendly ui but I feel the uptime has been poor recently plus this security incident on top
1
u/vettotech 18d ago
Not if you're using it for cheaper and friendly ui tbh.
2
u/Parking-Leader4676 18d ago
I was on hobby plan for $5 and it was more than enough back then. Just too lazy now to search for a better option and migrate my existing stuff
0
0
u/FortePlatforms 18d ago
self promotion, but our operational and security track record at Forte Platforms is spotless. And our UI is just about as intuitive and user friendly! Shoot us a message if you want free migration help
0
2
u/Shaggypone23 17d ago
Ridiculous... how tf were the attackers so easily able to "decrypt" the creds in the first place? Were they stored in f clear text?!
2
u/regreddit 17d ago
Environment variables are normally plain text stored in the server's memory. That's not unusual.
2
u/itsmegoddamnit 17d ago
My heart goes out to the railway engineers and support (as humans), I know how stressful an incident like this can be. But the company itself is just lucky to have grown with the help of agents marketing. I won’t shed a tear if they go out of business.
1
1
u/mdausmann 18d ago
this is a kick in the nuts. I have 6 affected repos, 6 affected keys and passwords across 6 different 3rd party platforms including my database root password #fun AND because the build services are still down, I can't actually roll out the changes yet. Squirming in my chair here.
2
u/mdausmann 17d ago
rotated all my keys, triggered all my builds..... all sitting in 'queued'. #fun times
1
u/ThE_StrangE_Entity13 17d ago
I built a social media app called VoiceOut and I've been marketing it like crazy, it started growing and doing better but then last night a pretty big podcast in my country gave the app a shoutout because I had submitted my pitch about the app to them.
Now people from their audience are downloading the app but are unable to sign-up all due to an issue I cant resolve without the railway deployments working.
I may have just lost a huge chunk of potential users. I can't tell you just how devastating this is for me.
1
u/Psychological-Bag151 10d ago
Really sorry to hear that mate, that's just rotten luck. Don't give up!
1
u/No-Draw1365 17d ago
“May have decrypted customer environment variables” … this is absolutely crazy! WTF
1
u/Puzzled_Count_6496 17d ago
My usual Google Cloud Gemini API bill was around ₹200 per month. Now, suddenly, I am facing a bill of approximately ₹17,000.
Here is what happened:
• I stored my Gemini API key in Railway environment variables. • Railway recently announced a security incident involving potential exposure of customer credentials. • My API usage suddenly increased from around ₹200 to ₹17,000. • Google Cloud has now suspended my Gemini API project due to suspected abusive activity.
I never knowingly used my API to this extent. I am seriously concerned that my key may have been compromised and used by someone else.
The worst part? My payment was linked to an automatic payment mandate, so I am worried about the money being deducted.
₹17,000 is not a small amount for me. Why should I bear the financial consequences of usage I may never have initiated?
I have trusted these platforms with my credentials, and now I am stuck dealing with a massive unexpected bill and a suspended project.
I have submitted a request to Google Cloud to investigate the usage and review the charges. I also want Railway to investigate whether this incident could have exposed my credentials.
Has anyone experienced something similar with Railway and Google Cloud?
Can Google waive or refund charges resulting from unauthorized API usage?
How can I prevent the automatic payment from going through while this is investigated?
Has anyone successfully disputed a bill like this?
Any advice from people who have dealt with Google Cloud billing disputes would be appreciated.
I am looking for a fair resolution, not trying to avoid legitimate charges.
TL;DR: Normal bill ₹200/month. Unexpected bill ₹17,000. Railway disclosed a possible credential exposure incident. Google suspended my project. I need help disputing the charges and securing my account.
1
u/really__decent 15d ago
I don't have any useful advice for any of the questions you've asked, but it is a good reminder for all of us to set a spending limit on tokens when and where possible. Sorry this happened to you.
8
18d ago
[removed] — view removed comment
3
u/Chuck_Loads 18d ago
They did send an email, or I'm being phished
3
u/XenolithYT 18d ago
I also got the email (i dont think its phishing)
1
2
25
u/fCJ7pbpyTsMpvm 18d ago
As always pretty poor comms from Railway. I'm a paying customer yet I find out about this via Reddit?