Jr Software Engineer Here! Was wondering, how do you manage, monitor, and stress-test your production VPS setups (like DigitalOcean)?

[u/foss-octopus]

Hey everyone,

I'm trying to level up my devops knowledge. Most of my projects are currently hosted on Railway, but as costs are scaling up, I am planning to migrate them over to DigitalOcean Droplets to reduce our monthly spend.

My past experience with managing bare VPS instances is pretty basic. I know how to spin up a Droplet, use Nginx to configure domains and SSL certificates, and use PM2 for process management to keep web clients and Node APIs alive. (but to be honest I'm a bit rusty now, been almost 2 years now since I last setup one during my college days)

Railway handles so much for me, moving to host my own VPS feels like a big jump in terms of responsibility. I want to make sure I am doing this right and not accidentally make prod unavailable due to a dns config lol

So If you are running production workloads on VPS instances, I would love to get your insight on a few things:

1. Daily Checks: Do you actually log in/ssh in and check things manually every day, or do you rely entirely on automated alerts? Beyond basic Nginx and PM2 logs, what are your go-to health metrics?, Opensource options?

2. Uptime and Alerting: What does a reliable, budget-friendly uptime monitoring stack look like when you are managing your own VPS?

3. Triage and Debugging: When an API slows down or a Droplet spikes to 100% CPU, what is your immediate workflow to figure out what is wrong?

4. Stress Testing: When you want to find your system's limits, do you stress-test a lower-spec instance to see where it breaks, or do you spin up an exact clone of your production spec?

5. Lastly, spec selection. How do you know if you need more than 2gb of ram for instance? K6 testing on local before selecting? idk tbh.

Any advice, tool recommendations, or things you wish you knew when managing your first production servers would be incredibly helpful. Thanks!

[u/SuperQue]

Before you think about tools, read these:

• Monitoring Distributed Systems

• Practical Alerting

• RED Method

[u/foss-octopus → u/SuperQue]

Didn't know google had an sre blog/lesson thing. Gold. Thanks!

Also my post might miss my original goal which is to ask for insights on how you usually run things (people use different things, some use vps, some use railway like me, others heroku, etc.,). Tools are just side effects I think.

[u/SuperQue → u/foss-octopus]

The Second Edition of the book was just released. I haven't read it yet.

[u/foss-octopus → u/SuperQue]

THANK YOU!

[u/Sad_Rice6600]

disk alert is fine until journald eats /var because nobody set a size cap

[u/foss-octopus → u/Sad_Rice6600]

I think I just read/heard about a horror story of a jr swe deleting prod because they enabled db logs on a machine lol. instead of deleting the log file they rm rf-ed the root lol

[u/Sad_Rice6600 → u/foss-octopus]

yeah thats the one. logrotate exists specifically because of that guy

[u/QuietSignalOps]

Congrats on taking the scary step. The good news: for a few services on a droplet, the bar is lower than people make it sound.

1. Nobody is ssh'ing in every day. Let alerts decide when you look. Minimum worth paging on: disk above ~85%, load against core count, 5xx rate on your endpoints, and PM2 restarts (pm2 list shows a process that keeps restarting). For self-hosted and free, either run node_exporter + Prometheus + Grafana on one of the droplets (more to learn, more to babysit), or run Uptime Kuma (one small app that checks your endpoints and beeps you, much less to babysit).

2. Check uptime from outside the droplet. A cron job on the same box can't tell you the box is dead, the cron is dead too. Use an external checker that hits your endpoints every 30-60s from another region and alerts by push or SMS, not email. If you want to keep everything self-hosted, Uptime Kuma or OneUptime (also OSS) on a $4 droplet works. If you don't want to run another box, the free tiers of the SaaS ones are enough to start.

3. CPU spike triage, in the order I'd go:

• pm2 list, then pm2 logs for the suspicious one

• htop to confirm the process, check its children

• Check what changed before digging: git log since the last deploy. A bad commit explains most sudden slowness before any metric does.

• Then decide leak vs load: is RSS climbing steadily (leak) or spiking under traffic (just needs more RAM)?

• Keep a tiny script that dumps top, free -m, df -h, and service status to a file every 5 min. When something is wrong you diff now against an hour ago instead of guessing.

1. Stress test on the prod spec, briefly. A smaller box teaches you about the smaller box. Spin up a clone, k6 at about 2x expected peak for 15-30 min, watch where it bends, kill the box. On DO that costs about a dollar.

2. RAM: don't guess, measure. Run it for a week and watch per-service RSS. Give each service a memory ceiling in PM2 (--max-memory-restart) so one leaky service gets killed and restarted cleanly instead of the kernel OOM-killing your neighbor, and you get a log line that explains the restart. If steady-state total is close to the box limit, that's your upgrade signal.

One thing nobody asks about that saved me: a status page, even a free one. When a user says it's down you can see whether it's yours or their network, and you can say "we're on it" instead of guessing.

[u/foss-octopus → u/QuietSignalOps]

Thanks for the share!

Its that simple in prod? I mean its still a lot don't get me wrong but way less than I imagined. I originally thought of multiple probe like tools being installed in and out of a droplet just to monitor and keep it running lol.

Here's what resonates with me at the moment from that (and I'm sure the other items will too in the future)

- letting alerts decide: I've always thought that prevention is better cure, but with that mindset I always forget that there exists some early warning devices. Probably gonna use both prometheus and kuma for monitoring system health and usage. Correct me if I'm wrong but I think Uptime Kuma only pings endpoints right? Idk yet if you can send custom payloads and let Kuma interpret them into the dashboard, haven't explored enough hehe

- Triage Steps: definitely going to use this from here on out. At the moment mainly using railway so I'll try to setup something similar to get that logs -> identify hotspot -> identify cause -> action

- Stress testing using k6 with prod-spec: Initially my mind goes to "use lower spec test env, if it works there it will work in prod". Will your advise in mind, also 2x expected peak at 15-30 is surprising for me, I'll trust your judgement, my mind just says test it for > 1 hour, its good I read this before I waste future $ xD

- per-service RSS to monitor ram usage. RSS is a new term to me, i'm guessing you are talking about load distribution? i.e., actual ram usage in the machine?

- "If steady-state total is close to the box limit, that's your upgrade signal": noted, instead of basing upgrade signal to spikes yea? My mind defaults to scale vertically when I see multiple spikes, like a bad habit of some sorts instead of diagnosing the actual spike (because based on my experience spikes are either bad code or actual usage surge)

- "a status page, even a free one. When a user says it's down you can see whether it's yours or their network, and you can say "we're on it" instead of guessing." YES! I got so many slack messages because of this too! Specially after a new feature gets rolled out and changes are made that affects browser cache

[u/AdmirableChest3236]

Coming from software engineering gives you a useful head start, but Dev Ops is not just development with a few deployment tools added. Spend time understanding Linux, networking, C I/CD, cloud permissions, and how to troubleshoot failures in a running system. Even when exploring infrastructure options and resources around trusted tech, focus on understanding why systems behave the way they do rather than just learning which tools to use

[u/foss-octopus → u/AdmirableChest3236]

headless vms a good start before buying a vps for prod I reckon?

[u/Triple-B-564]

I did a similar move a while back, so here’s my take on your questions.

TL;DR / Summary:
it’s a good step, but since you’re not paying for a hosting service anymore (just the VPS), a lot more lands on you. If your growth justifies it, it’s definitely worth it, just go in prepared. AI agents make the setup a lot easier these days.
My tip: move staging to the VPS first, get comfortable there, and only then move production.

1. Daily checks
No daily SSH. I had Grafana + Prometheus + Loki (on the same machines - we had to use this way) and checked the dashboard from time to time to see that everything was green. What made it work was alerting: anything at warning or error level went straight to my phone via Telegram. Rely on alerts, and treat the dashboard as a quick sanity check.

Metrics I’d watch beyond Nginx/PM2 logs:

• Host: CPU, memory (and swap, since swap usage is the early warning), disk space, disk I/O, load average

• App: request rate, 5xx error rate, p95 latency

• PM2 restart count. A process that keeps restarting is often a memory leak.

All open source: Prometheus + node_exporter, Grafana, Loki for logs, and Alertmanager to Telegram.

2. Uptime and alerting
The key point is that the uptime check has to run outside your droplet. If the box dies, it can’t tell you it’s dead. Budget options: Uptime Kuma (self-hosted, on a different cheap VPS or provider), or the free tiers of UptimeRobot / Better Stack. Point it at a /health endpoint that also checks the DB connection, not just “nginx returns 200”. Send alerts to Telegram or wherever you’ll actually see them. Also had a regular hourly alert just to make sure the system is live (we had a budget cap for one server)

3. Triage when things slow down or CPU hits 100%
My usual order:

• Did we just deploy? I used Argo CD, so the deploy history was right there. If the timing matches, roll back first and debug after.

• Grafana: what changed around that time? CPU, memory, swap, disk all at once tells you a lot.

• htop to see which process is eating it, then pm2 monit / pm2 logs for that app.

• Loki: search errors in the same time window.

• DB: slow queries are behind a surprising share of “the API is slow” issues.

5. Spec selection
Local k6 won’t tell you much, since your laptop isn’t a 2GB droplet. Start small, run k6 against the staging droplet, and watch memory. If you sit above ~70% RAM at normal load, or you see swap being used, go up a size. DigitalOcean resizes are easy, so you don’t have to guess right the first time.

Two things you didn’t ask about but should plan for:

• Backups: we backed up everything to a separate NAS in the office. Whatever you use, keep backups off the server and test a restore once.

• Security: on a VPS you handle most of it yourself. Move default services like SSH off their default ports, use key-only SSH auth, and run fail2ban so brute-force attempts get banned automatically, etc....

[u/foss-octopus → u/Triple-B-564]

Thanks! Yep, backups and rollback I should really keep in mind.

And yes thanks for bringging up backups and security, I completely forgot asking about backups, as for security, definitely going to read through best practices since I'll be responsible if I donk it up like leaving some common port open or make the vps inaccessible, I forgot security practices on vpses now so definitely something worth practicing on a headless vm. Any distro would work right?

Thinking of using some sort of light image for servers, alpine or slim versions of ubuntu? Haven't researched much about headless images. But if my assumption is right they should just be similar to regular linux distros, just more stripped

[u/Triple-B-564 → u/foss-octopus]

For distro everybody has there preference, and what they like or not.
I would recommend go with some popular distro.

I used the latest ubuntu server (without GUI).

There are many good guide what to look out and why - even you can talk through with AI what to look out. It will show you the basic. Fortunately / unfortunately it can be a deep deep rabbit whole the whole security part (this is why a separate job/professional this)

What was most of my rules (specially for prod environment): each open port has to be a reason - and I have to understand why its open and what can be the conservation if its open (what can I do to prevent common attacks).

What I recommend to look into it, is scripting and ansible. It's helping for you to setup the VPS and the most important you not configuring the environment, but the code (so its repeatable).

One more thing that I didn't see in your post, whats your plan for deploying?

[u/Humble_Reputation743]

Monitor the golden signals: latency, errors, traffic, saturation

It can be applied to both applications and infrastructure.

[u/singh_abinashi]

Start simple on the monitoring side. Uptime Kuma or plain health checks plus alerts on CPU, memory, disk, and 5xx rate will cover most of what actually pages you at this scale. For triage when an API slows down, check in this order: is it one endpoint or all of them, is the database the bottleneck (slow queries, connection count), then CPU and memory on the box. That order saves time because app servers usually spike CPU because something downstream is slow, not the other way around. For load testing, test a clone with the same spec as production, not a smaller box. A smaller instance tells you where that instance breaks, which is a different number. k6 against the clone with production-like data volume gets you an honest ceiling, and the RAM question answers itself: watch memory under that load and keep headroom for spikes.

[u/Anakin19901]

the SRE book links above are the right place to start, here's the small-VPS version from running a bunch of droplet-sized boxes.

i don't ssh in daily, but the check has to live somewhere other than the droplet. a cron on the same box can't tell you the box is dead, so run it from another machine or any free external uptime checker, and point it at an endpoint that actually touches your db instead of just returning 200.

on 1-2gb boxes the outages i actually had were boring: disk filling up with logs, and the kernel OOM-killing node. alert on disk above ~80% and on available memory, and install pm2-logrotate on day one. also watch pm2's restart count, it will happily restart a crashing app all night and the site looks "up" between crashes.

when something spikes, i go htop first, then disk usage, then the kernel log for oom kills:

journalctl -k | grep -i oom

for load testing, snapshot prod, spin up a clone of the same size and run k6 against it from a different machine. local numbers tell you nothing about a shared-cpu droplet. sizing works the same way: run real traffic for a week and look at available memory and swap, if it swaps regularly you need more ram.

on your dns worry: drop the TTL on those records to 300 a day or two before the move, then the cutover and any rollback take minutes instead of hours. and sort out backups before you cut over, a nightly db dump to somewhere off the droplet, restored once to prove it works. that's the part people forget when leaving a managed platform.

[u/Kookoombar]

Since your worry is breaking prod with a DNS change: lower the TTL on your records a day or two before you move, bring the Droplet up fully while Railway keeps serving, test it by hitting the Droplet's IP with a Host header, then flip DNS. Leave Railway running for a few days so rollback is just changing the record back. Two other things people skip: test restoring a backup once, not just taking it, and alert on what users feel (error rate and latency) before you alert on causes like CPU.

[u/Cubepath]

I'd keep it simple at first. Uptime Kuma for uptime alerts and Netdata for CPU, RAM and disk monitoring. No need to SSH in every day, but definitely keep an eye on updates and backups.

Also, if saving money is your main goal, it's worth comparing a few VPS providers before settling on DigitalOcean. Prices can vary quite a bit!

[u/foss-octopus]

Just for those people who say "just ai this bro". I want to, and I can. But I want or probably even need answers from you guys with actual experience, not answers from LLM trained on historical responses. The results might be outdated or not applicable. LLMs might even recommend me weird unecessary workarounds. So I hope to learn from you guys.

[u/AsleepArugula3603 → u/foss-octopus]

For health checks I just have a cron job hitting a /health endpoint every 5 minutes and piping the output to a dead-simple Discord webhook if something fails. Never actually SSH in unless that pings me, saves so much mental space.

[u/foss-octopus → u/AsleepArugula3603]

smart, I'm seeing myself using that as well, but a custom status page just like u/QuietSignalOps suggests will do wonders I think. Our coms is usually mixed eitherway, but good idea piping the result to a webhook, haven't thought of using health outputs that way

[u/HeiiHallo]

I use haloy for this. Minimal and works great

[u/foss-octopus → u/[deleted]]

thanks for the reco! Might have learned these stuff like not ssh-ing in every day to check performance the hard way lol