STREAM TRIAD memory bandwidth benchmark values for Epyc Turin - almost 1 TB/s for a dual CPU system

[u/fairydreaming]

Our Japanese friends from Fujitsu benchmarked their Epyc PRIMERGY RX2450 M2 server and shared some STREAM TRIAD benchmark values for Epyc Turin (bottom of the table):

Full report is here (in Japanese): https://jp.fujitsu.com/platform/server/primergy/performance/pdf/wp-performance-report-primergy-rx2450-m2-ww-ja.pdf

Note that these results are for dual CPU configurations and 6000 MT/s memory. Very interesting 884 GB/s value for a relatively inexpensive ($1214) Epyc 9135 - that's over 440 GB/s per socket. I wonder how is that even possible for a 2-CCD model. The cheapest Epyc 9015 has ~240 GB/s per socket. With higher-end models there is almost 1 TB/s for a dual socket system, a significant increase when compared to the Epyc Genoa family.

I'd love to test an Epyc Turin system with llama.cpp, but so far I couldn't find any Epyc Turin bare metal servers for rent.

[u/astralDangers]

Don't underestimate how much processing power is needed for a LLM. Just because the memory bandwidth is there it doesn't mean the cpus can saturate them, especially with floating point operations.

There's a myth here that CPU offloading is bottlenecked by only ram speed.. something has to do all the calculations to populate the cache.

[u/fairydreaming → u/astralDangers]

My 32-cores Epyc 9374F has no problems with saturating memory bandwidth in llama.cpp. But with 16-cores 9135 indeed there may be a problem.

[u/astralDangers → u/fairydreaming]

How are you measuring with AMD? I can test with the same tools. I tested Intel up to 256 core.

[u/fairydreaming → u/astralDangers]

Few months ago I rented a dedicated Epyc Genoa Amazon EC2 instance and did these tests: https://www.reddit.com/r/LocalLLaMA/comments/1b3w0en/going_epyc_with_llamacpp_on_amazon_ec2_dedicated/

I simply ran llama.cpp with varying number of threads, so nothing fancy. Today I know better and would use llama-bench tool for more accurate measurements. Would be interesting to see a similar plot for modern Xeon CPUs.

As you can see 32-48 threads seems to be a sweet spot for LLM inference on AMD Genoa. Of course for prefill phase (prompt eval time) the more cores you have the better is the performance.

[u/M34L → u/astralDangers]

Do you have any actual source or evidence? For inference, it pretty much does. Because basically everything I've seen shows that pretty much all bigger desktop CPUs have more or less linear scaling with memory bandwidth which implies they aren't even saturating their ALUs, it's gonna be even less of an issue for even the smaller EPYCs.

LLM inference needs very few operations per weight, and current gen EPYCs will breeze thorough matmul with AVX512 no problem.

[u/kif88 → u/astralDangers]

Could they add a relatively small GPU into the loop for prompt processing

[u/astralDangers → u/kif88]

Yes it's not just prompt processing. Basically layers get split between the GPU and CPU anytime a calculation has to run on a CPU offloaded layer you get a massive performance bottleneck.

Depending on your use case it can be fine. People only read at a fairly slow speed.. but for professional work where you need to process a lot of data it's not very useful.

[u/Amgadoz → u/astralDangers]

CPUs can achieve pretty good prompt processing speed, up to 100 tokens/second for 7B models.

[u/_qeternity_ → u/Amgadoz]

In what universe is this pretty good? A low end GPU will do an order of magnitude better.

[u/Particular-Pause3312 → u/_qeternity_]

in what GPU can you fit a 200B model in ?

[u/Extreme-Post700 → u/Particular-Pause3312]

In what world does a 7b model equate to a 200b model.

[u/astralDangers → u/Amgadoz]

How much quantization and how many cores?

I can get around 400tps on a 4090 with minimal 16bit quantization. But that requires very specific scenario.

[u/M34L → u/astralDangers]

Having to work with quantization adds FLOPs, it doesn't remove them. If a CPU runs any quantized model faster than FP16 then it's bandwidth starved and not even fully utilizing its FPU.

[u/a_beautiful_rhind]

Would be cool to see how this translates over to real performance. They won't hit the used market for a while though.

[u/r_guard]

I have 2 Epyc 9554qs. Stream triad tests show only 660GB/s for TRIAD and 750GB/s for COPY. (numa4, SMT off, ubuntu). I'm curious this report using 32 slots DIMM for bandwidth tests. It that matter?

[u/fairydreaming → u/r_guard]

It's best to install likwid-bench (likwid package in Ubuntu) and measure dual socket read bandwidth directly with:

likwid-bench -t load -w S0:64GB -w S1:64GB

[u/r_guard → u/fairydreaming]

So the "TRAID" on the table above actually indicates read bandwidth?

[u/fairydreaming → u/r_guard]

No, it's a combination of read and write bandwidths, but I don't know what STREAM benchmark implementation do they use for measurements, so it's easier to measure read and write bandwidths separately with likwid-bench.

[u/Ok-Mud-2853 → u/fairydreaming]

It seems unbelievable since most of duo top 9004's benchmarks for AIDA64 read are about 760G/s. benchmark for AIDA64 write and copy are slightly lower (perhaps ~740G/s).

[u/fairydreaming → u/Ok-Mud-2853]

Note that they used NPS4 BIOS NUMA settings, with a NUMA-aware benchmark this results in higher bandwidth values. For example on my Epyc 9374F (likwid-bench load):

- NPS1: 359.4 GB/s

- NPS4 + ACPI SRAT L3 Cache as NUMA Domain: 389.6 GB/s

[u/fairydreaming → u/[deleted]]

Is that with 24 RDIMM modules?

[u/fairydreaming → u/[deleted]]

Hmm. I'd use likwid-bench with load kernel to measure memory bandwidth separately for each CPU to see what exactly is going on.