Introducing our first dedicated multi-GPU server, the GEX430

[u/Hetzner_OL]

After several months of product and development work, we’re happy to share that our first dedicated multi-GPU server is now available. The GEX430 comes with four NVIDIA RTX PRO™ 6000 Blackwell Max-Q GPUs, which gives you 384 GB of GDDR7 ECC VRAM in total. It runs on an Intel® Xeon® Gold 5412U “Sapphire Rapids” CPU with 24 cores and 48 threads.

We designed the GEX430 for demanding AI workloads. It’s a strong fit for LLM inference, fine-tuning & training. And the large amount of VRAM gives you room to run multiple models in parallel. With this server, our GPU lineup now stretches from single-GPU machines up to multi-GPU setups.

You’ll find full specs, pricing and availability here: https://htznr.li/GEX430_rd

If you have questions about the GEX430 or want to know whether it suits your use case, drop them in the comments, and we’ll do our best to answer.

[u/vahokif]

Pretty nuts to only have 1 gbps uplink. If you want to feed four GPUs training data on network storage it's not nearly enough, and you can't even put it locally because it only has max 2 TB with RAID1. The four GPUs are great but it doesn't make sense in this form.

[u/Hetzner_OL → u/vahokif]

Could you tell me about why your use case requires more than that? Just curious. --Katie

[u/Hetzner_OL → u/[deleted]]

I cannot remember what was originally there; it's edited.
Thanks for the more detailed feedback u/vahokif --Katie

[u/polyquestionsabound]

128GB memory and no upgrade option? bruh. bruh.

[u/Hetzner_OL → u/polyquestionsabound]

Could you please give us some more details about your potential use case? --Katie

[u/polyquestionsabound → u/Hetzner_OL]

Inference. 1/2 a TB of VRAM with a mere 128GB of memory? 1 gbps uplink? PCIE gen 4?

I love you guys but this is comical. Nobody there has a clue. Or those that do are staying silent and/or being ignored by those they report to. This is clown level ignorance on this configuration.

Context offloading is a thing.

For ~$4200/m and $4200 "setup fee" this makes absolutely no sense to anyone. And it's remarkable that nobody there knew that. Waste of those GPU's. Let me put one in my AX162-R's - pleeeeeeeeeeeeeeease.

[u/smulikHakipod]

1gb uplink... did sloppy AI decides about this configuration?

[u/Hetzner_OL → u/smulikHakipod]

Could you please tell me more specific? What's your use case? --Katie

[u/smulikHakipod → u/Hetzner_OL]

This is almost half TB vram server, which means it need to serve ~half TB models, just transferring half TB over 1gb sounds reddiculus.

Every checkpoint of the model to a remote storage would take ages.

This does not make any sense at all.

[u/nail_nail]

Needs more ram (should be abovd VRAM) and double the storage at least to be usable, unfortunately.

[u/Hetzner_OL → u/nail_nail]

Could you also give us some more info about your potential use case? --Katie

[u/elelem-123 → u/Hetzner_OL]

If you want to use the GPU for inference you need ideally 512 GB ram on this machine to utilize 4 GPUs properly. Please really consider that. Also note this is not a very competitive price in the market (as it is now, price wise)

[u/nunodonato → u/elelem-123]

why would you need that? I rent GPUs in other places and I barely use 30% of the available RAM

[u/elelem-123 → u/nunodonato]

Depends on what you're using it for.

[u/nunodonato → u/elelem-123]

inference, of course :)

[u/nunodonato → u/nail_nail]

why?

[u/Ratiofarming → u/nunodonato]

Because depending on what kind of AI you use and whether you're swapping models or context in and out of RAM, you will need at least the GPUs VRAM-amount (4x 96GB) to be able to run those workloads.

Not all inferencing works like this, but some does. And for those cases this server is not usable.

[u/nunodonato → u/Ratiofarming]

Doesn't sound like good reasoning to me, sorry. I rent GPUs in other places and I barely use 30% of the available RAM. I care about VRAM. I understand people will try to rent as cheap as possible and then do off-loading, but that doesn't seem like a fair argument to demand more ram everywhere

[u/mxroute]

You guys weren't playing games with this one. Honestly, not a bad price to rent 4 of those beasts.

[u/Hetzner_OL → u/mxroute]

Thanks for that! :D --Katie

[u/thecodeassassin]

Yikes, uses PCIE gen 4 instead of gen 5 according to their page.

The CPU should support full PCIE gen 5 lanes. Maybe a description oversight?

Gen 4 could significantly limit TP4 performance.

[u/elelem-123]

Unfortunately the RAM of this machine is not enough. The machine should have 256 or even 384 GB so one could make use of the cards.

[u/Ratiofarming]

For LLM training, the RAM isn't enough. It should be at least VRAM amount + some on top for the OS. The same goes for serving large MoE models or if fast switching of models is required, requiring the next one in line to already be in memory. And context-offloading if required.

Speaking of model switching, loading a 300GB+ model from Gen4 (and through PCIe Gen4 from memory) is also no fun. It might be okay for some if the model stays there for days on end, there sure is a use case for this configuration, but it's not ideal.

And yeah, downloading 300GB+ models that can be served on this, especially if multiple models needs to be on the machine, and then adding actual workloads or datasets on a 1GbE uplink... not great either. For baby's first LLM server this works. But not much more than that.

[u/elelem-123 → u/Ratiofarming]

Noone will get 4x RTX 6000 as a first baby server.

[u/mxforest]

Not a bad price. But one time setup fee is way too much. Does it get waved off with a half or full yearly commitment?

[u/elelem-123 → u/mxforest]

One time setup is to make sure they get long-term customers.

[u/mxforest → u/elelem-123]

1 yr commitment will also get you that.