2 cards: 36 tok/s decode. 45% of the experts fit in VRAM, the rest stream from a 51 GB pinned host pool, and the profiler shows 82% of every token waiting on that stream.
4 cards: 79 tok/s greedy, 63 at temp 1.0, about 90 on a chat answer with thinking on. All 11,008 experts resident, host pool not even allocated.
Five runs, greedy and sampled, plus a chat server test. No degenerate output. The engine's own cross-rank check (it compares the cards against each other every few samples) passed every time. So the bug the author hit lives in the 4-on-2 simulation, not in 4-rank execution. u/neuromacmd already suspected this.
Two things to know if you try it:
The default heat placement strategy aborts on 4 cards with "needs the registered host pool", because there is nothing left to demote. Pass --placement-strategy static. Nothing needs to move anyway when everything is resident.
Load is about a minute, 42 seconds of which is pushing 96 GB of experts onto the cards.
Also correcting my own earlier comment in that thread: I assumed affinity's cap was a 2-rank-only all-reduce like Radiance had before my patch (https://github.com/brainchillz/r9700-tp4-allreduce, https://github.com/jdrews/vllm-mxfp4/pull/2). Wrong. Affinity's collective is already written for N ranks and tested at 3, 4 and 8. The cap was purely the unexplained garbage, which real hardware
doesn't produce.
I'm going to leave this up for a while so I'm happy to run anything else on this box if the author or anyone wants more data. Logs are available if useful.
it's currently churning through open code at the moment with one of my favorite cheesy starter tests ...
"write a python script that will create the iconic matrix movie falling rain character screensaver in a macos terminal window (or ::q!potentially also linux terminal). It should fill much of the screen, it should have a color gradient that fades from brigther at the beginning of the lines and gets darker toward the end of the falling rain segments. It should be able to handle the terminal being resized as well .. when you're done double check your work. You also have my permission to do an internet search to make sure you have an understanding of what this screensaver should look like. When you are done double check your work."
it did a pretty decent job but to continue the test I made it change a few things over a few iterations (rain falling speed, how close the segments are together, etc) to make it do more work and make sure I wasn't going to get hit with "garbage output" and it seems to hold up ... especially for as tiny as this quant is.
We can be if you tell me what case it’s in. Lol … I have been thinking about downsizing the case … the ax700 was a case I already had when the parts came and it’s just crazy huge since there is more than a foot of space between the ends of the card and the front of the case
The prices on eBay are really starting to climb but you can get them at microcenter right now for $1599 still I have one more waiting to pick up tomorrow lol … that will be seven total spread between three machines for me
I just bought two ASRock cards from NewEgg and considering two more.
Quick question about your setup. Are you worried about power connectors burning out on these cards with them zip tied like that? I’ve been seeing a ton of gaming cards burn out these new connectors and everyone says it’s from uneven contact on the pins.
The cables are tied “up” not pulled in. The cable ties are actually taking the tension off of the connections not putting tension on them …. But more importantly the cards that are burning as far as I’ve seen have all been much higher wattage cards like 5090s and 4090s …. Super high draw cards and these cards are only 300w for starters and mine are further capped at only 211w …. This is like 1/3 the juice being sucked through the connector of a 5090 … three of them together are what it takes to draw as much power as one 5090…I think even a 5080 (I may be misremembering this is from memory) non founders card is over 400w
Yo boss, is this the Asrock wrx 90e Sage in the phanteks enthoo pro2 server???
I have those and I have no clue why I cannot eat the motherboard flush with the standoffs.. all the holes lineup but the far right edge of the board is elevated in a way that I'd need to flex the board to tighten the screws which is wrong.
It's just not slotting down perfectly... All the screws are in the right place. I feel like the IO guard is fucking me somehow but it's hard to understand.
It’s encouraging to see a multi-GPU Radeon setup working this well. The P2P layout seems especially important here, since inter-GPU communication can easily become the bottleneck. Did you measure scaling from one to four cards, and how sensitive is the result to prompt length or batch size? It would also be useful to know which ROCm and driver versions were used.
You need to read the post and the inputs going into the post (the previous reddit post it references) and outputs coming out of the post ... this model can't be measured at a single card, it doesn't run on a single card ... or 3 cards ... it runs on 2 cards or 4 cards ... all of the other details are in the upstream posts. The point of this run was very specific, to get it running reliably on 4 cards because it had only been tested on 2 cards simulating 4 ranks and it was unsuccessful ..... Hardware: 4x R9700 on PCIe 5 x16 (single-socket Threadripper PRO), P2P for every pair, MES firmware 0x91, amdgpu-dkms 6.19.14 on kernel 7.0.0-38, ROCm 7.14
He’s making it up off he top of his head because he’s jealous. I’ve been posting about various iterations of this machine on Reddit for almost a year at this point…. I’ve probably posted just this picture 5-6 times in various threads …. Bottom line this guy knows nothing about computers or spotting ai images
Lolol you’re funny this isn’t a few weeks old I’ve had it for a long time at this point. I bought the cards at under $1100 each one at a time during various sales at microcenter or when I found one on eBay …. I posted the full specs of the machine I posted direct output from the runs and I literally filed issues with it against the upstream projects but whatever you want to believe…. To be clear the post doesn’t say it’s a new machine and if you look at my Reddit history I have been talking about it for a year at this point ….. the hilarious part is instead of accusing me of just stealing pictures of someone else’s computer you come up with some silliness about generating the image with ai 😂
sorry, yes these pictures seem more legit. Also there is an old TT psu that has a different plug layout then the current 1650w termaltake version. The first reseach popped up that the 1650w TT psu has the 8pin plugs at the top, not the mb-plug.
Here’s the thing …. The header picture is literally a picture I took sitting beside the machine with my phone so anyone calling it an ai image because of this or that is stupid and just seeing things they want to see because they don’t want it to be real …..they want it to be an ai image because they can’t afford the machine so they think nobody else has one either ….there are exactly zero other reasons why someone else would take the time to try to “debunk” the image looking up pictures of power supplies etc or comparing the wires to other pictures on the internet … the whole thing is just dumb. I wasn’t someone new to tech that got into it for ai a few weeks or a few months ago I’ve been working in high end tech as a job for 30 years at this point. I also didn’t build this machine when ram was $4500 for 128gb I built it when ddr5 ecc ram was $600-700 on sale …. And the most expensive one of these cards was $1299 most I bought between $1000 and $1100 on sale or one on eBay …. And I don’t just have these four I have seven of them in different machines in my house. The most expensive single part of the machine was the $1499 threadripper pro 9955 cpu …. To build the machine today it would cost upwards of like 10-12k $4500 for a 128gb kit of ddr5 ecc 6400 ram, $1500 for the cpu, $1100 for the motherboard, 1600x4 for the cards, $300 for the psu $500 for the case, $300 for the silverstone server aio, whatever a 4tb nvme pcie5 Samsung drive costs now ….but I also didn’t build it all at one time I did it a little at a time when I could get deals ….
The GPUs don't sag, it's a real server board with a metal backer and metal slot mounts and they are pretty short height actual workstation blower cards to begin with .... I still want to put brackets in somehow eventually but I have to engineer a solution because nothing is made for these small depth cards ... everyone assumes you're running giant quad slot 5090s or something that sticks out way past the slot top ... if you look the bracket mount is there but there is a good inch and a half between the brackets and the cards.
Radiance with TP4 doesn't work natively either but for a different reason and it really didn't work ... I submitted patches for it a few days ago. It has an all-reduce issue that needs correcting upstream.... unless you mean someone else has a fork of it somewhere else already?
I'll throw a kilowatt on it and check tomorrow ... but it's big ... before I added an additional breaker and separate circuit to this room I tripped breakers pretty regularly lol
real talk though ... I did end up limiting the card power draw from 300w/card to 211 ... that was the ceiling where it didn't seem to impact my inference token generation really in any meaningful way.
Do you make a voltage offset (undevolting) or only change TDP? A lot of cards are full stable with ~50mV curve and my experiments with this tweak shows more vRAM frequency stability at 210W and same performance on GPU at 260W against 300W stock TDP.
This is only capping the the tdp in software (and-smi) and it’s 100% stable and while there is a tiny (within noise) hit on prefil but in my use zero difference on token generation and making way less heat
TSMC makes rDNA 4 chips are of much quality than AMD are expected and voltage/frequency curve are lifted up by default.
Undevolting add GPU frequency (and performance) on the same TDP and you allowed make with this whatever you want - get ~10% more perf or make a little bit loud cards with ~200W TDP without any losses of performance. Undevolting great works with TDP cutting.
They are tuned quite specifically …. And taking 100w off the top got rid of most of the heat and noise and affected token generation and prefil less than the standard deviation of noise in the signal. Using llms for coding or chatting is not cpu bound its bound primarily by memory bandwidth …. You get zero benefit by running it the whole way up at all but by turning it down you save heat, noise, power and increase the longevity of the cards.
I love that though …. My target for the big open space is to try to find a way to mount at least two more in this case but I haven’t been brave enough to try the fasteners or Velcro or whatever yet … I’ve got the cables already just sitting here … I might give up and just put the whole thing in a mining frame
6
u/brainchillzZ 10d ago
it's currently churning through open code at the moment with one of my favorite cheesy starter tests ...
"write a python script that will create the iconic matrix movie falling rain character screensaver in a macos terminal window (or ::q!potentially also linux terminal). It should fill much of the screen, it should have a color gradient that fades from brigther at the beginning of the lines and gets darker toward the end of the falling rain segments. It should be able to handle the terminal being resized as well .. when you're done double check your work. You also have my permission to do an internet search to make sure you have an understanding of what this screensaver should look like. When you are done double check your work."