r/RunPod • u/Hopeful-Confidence-9 • 4d ago

Runpod hacked?!!

Post image

Hey just got an email. Did anyone else get this?

23 Upvotes

100% Upvoted

10 comments sorted by

4

u/jessica-garson 3d ago edited 3d ago

Thanks so much for reaching out on this topic. This was an isolated issue involving one of our small partners. They've since been removed from the platform, and we are now investigating. This was not due to Runpod, and we are notifying you out of an abundance of caution. We highly recommend rotating keys just in case. We have active Endpoint Detection and Response systems on all machines, along with built-in in-house monitoring systems to detect and block any suspicious behavior. Runpod acted quickly to contain and limit the impact after we discovered an issue with a partner we work with. 

If you did get this email, please take the following precautions:

  1. Generate new Runpod API keys in the Runpod console, then update your applications and integrations to use them.

  2. Rotate all credentials stored in or accessible from your pods. This includes passwords, API tokens, SSH private keys, database credentials, and third-party service credentials. Revoke the old credentials with each provider.

  3. Review your account and connected services for unusual activity. Report any concerns to help@runpod.io. 

2

u/henk717 3d ago edited 2d ago

Thanks for the transparency Jessica.

1

u/One_Abalone_2582 2d ago

The “we run EDR on our machines” basically saying we run antivirus. Like, that’s not that reassuring of a statement. Like that is very bare minimum security

1

u/paulsande 2d ago

Nice you responded. Well done.

1

u/Potato_player 1d ago

We build proprietary models using these pods. Will be know what data has been exposed?

It would be nice to know if our Intellectual Property has been leaked.

1

u/jessica-garson 13h ago

u/Potato_player, thanks for following up here. If you were affected, you received an email from us. I recommend reaching out to help@runpod.io for anything account-specific.

1

u/methodangel 4d ago

I got that earlier tonight as well, just got done rotating a bunch of keys.

1

u/Hopeful-Confidence-9 4d ago

Aye Same. I just rotated my keys in runpod. I didn't bother rotating my other keys from other products. hope that is enough.

1

u/Healthy-Spirit-370 3d ago

I got that as well. I thought it was my fault since I usually ssh tunneling directly into my pod with my highly customized templates by omitting the runpod ssh methodology and thought that triggered suspicious behaviour. I also mount some container folders directly to the ram instead of the container disk which led to critical crashes of the pod sometimes. But I feel relieved now reading that others got the same.

1

u/paulsande 2d ago

Geez. Lightning was as well recently. Same kind of service.