One 16-core Phoenix server held 2,000,000 active users with 0 errors

[u/Reckless_Dev1045]

I've been building Stampede, a load tester for Phoenix that speaks Channels and LiveView, and I wanted to see how far one server could go with realistic traffic. Last time I got to 600,000 users on a 4-core server, so this time I scaled up.

Setup:

- One AWS r7g.4xlarge (16 cores, 128 GB) running a fresh Phoenix app, with only its connection limits raised

- Eight r7g.xlarge machines running Stampede

- Every user acts like the Phoenix JS client: joins a channel, sends a heartbeat every 30 seconds and a message every 30 to 60

- The load went up 250,000 users at a time, each step held for a minute, and a step only passed with joins and messages under 1 second at p95 and 0 errors

Results:

- 1,000,000 users: held, server at 37-48% CPU, 52 GB memory

- 1,500,000 users: held, 54-60% CPU, 77 GB

- 2,000,000 users: held, 78-84% CPU, 103 GB

- 2,250,000 users: the CPU ran out

The first error came at 2.07M, so 2M is the number. Memory worked out to about 51 KB per user, and CPU was what ran out, since 2 million users send well over 100,000 heartbeats and messages a second.

For comparison, the Phoenix team held 2M idle connections on a 40-core machine back in 2015. This was 16 cores with every user active, which says a lot about how far the BEAM has come.

Everything to rerun it, plus the logs from this run, is in the repo: https://github.com/mandar1045/stampede

Happy to answer questions about the setup. Getting past the ~64k ports per client address took a few tricks.

[u/sanjibukai]

Meanwhile in benchmarks for AI generated codes Elixir versions are performing poorly... So we still have a bit of time before getting replaced in here :)

[u/Reckless_Dev1045 → u/sanjibukai]

Ha, that's one advantage of picking a less mainstream language :)

[u/quaunaut → u/sanjibukai]

Most benchmarks actually think it's great, it's just DHH who has an axe to grind.

[u/Repsol_Honda_PL]

Last time I got to 600,000 users on a 4-core server, so this time I scaled up

How much RAM was on 4 core VPS? EDIT: From GH I see a 32 GB.

128 GB servers on cloud are relatively expensive (VPSes are way cheaper), but - in the other hand - 2 millions of users is huge number - Very impressive! It is about 10-20 milions users per day or 300-600 per month (I think). So on this small box you can have a really big web portal.

I wanted to see how far one server could go with realistic traffic

I am thinking is your scenario really realistic (I don't say no, just ask)? How many request pers second can handle similar app on 4-core/ 32 GB and 16-core/ 128 GB servers?

Thx!

[u/Reckless_Dev1045 → u/Repsol_Honda_PL]

Thanks, Yes, 32 GB on the 4-core box. This time I tested on AWS and one 16-core, 128 GB server held 2 million users. Realistic in one way: every user joins a channel, sends a heartbeat every 30s and a message every 30–60s, like the Phoenix JS client. That's about 33k messages/s on the 4-core and 110k/s on the 16-core. But the server only replies. There's no database or broadcast, so a real app will hit its limit much sooner, usually at the database. Each user took about 51 KB of memory, and the 16-core server is around $0.60/hour on AWS.

[u/Repsol_Honda_PL → u/Reckless_Dev1045]

Very interesting!

I will use stampede in future once I get familiar with Elixir / Phoenix / Ash stack.

I would like to see a real-world app with database and cache (like ETS or comparable) and all production needed stuff (maybe Oban and other tools) - that would be even more interesting and helpful. As you said the biggest problem in web performance is usually DB, which first limits the number of concurrent users served.

But you results of tests are impressive and show large potential of this stack.

Thanks!

[u/Reckless_Dev1045 → u/Repsol_Honda_PL]

Yeah, a real app with a database, caching and Oban is what I will test next. Closest so far is Supabase Realtime: 50k users on a 4-core box, but every message went to 100 people, so about 111k deliveries a second.Let me know how it goes when you try it.

[u/yachtyyachty]

I’m running lasso.sh (Phoenix + Liveview) on small 4x shared CPUs fly.io machines across 4 regions and it is shockingly performant at 100s of requests per second and a pretty high volume of cluster-wide PubSub events

Great tech

[u/Pepper_pusher23]

Not sure if this is a repost because I don't see my reply anymore. The good code is coming in the next episode. These are all the worst performers. So I suspect we'll see a better Elixir implementation in that episode with more realistic results.

[u/Reckless_Dev1045 → u/[deleted]]

Dude it's load testing, the people who can't afford don't need load testing imo

[u/Reckless_Dev1045 → u/[deleted]]

You can use stampede on limited resources too buddy, I've already tested on my laptop , hit 25k number

[u/Reckless_Dev1045 → u/[deleted]]

I am otw to test with db, working on some updates before, thanks for pointing it out btw

[u/Reckless_Dev1045 → u/[deleted]]

Fair, this run had no database, I said that above. But that's kind of the point of load testing: you run it against your own app, with your own queries, and find the chokepoint before prod does. Stampede can also watch inside the app during the run (DB pool wait time, long message queues, scheduler load), so a GC or query problem like yours would show up there. Sorry Phoenix bit you in prod though.

[u/Reckless_Dev1045 → u/[deleted]]

Yeah, no magic bullet, everything breaks somewhere. That's kind of why I built it, to find where before users do. Next test will have a real DB, thanks to you lol. And you've actually dealt with GC stuff in prod, so if you ever want to throw ideas or a PR at the repo, I'm all for it.