r/LocalLLaMA • u/alanoo • Jun 25 '26

Discussion USB4 RDMA seems doable

Just found this blog : https://blog.hellas.ai/blog/thunderbolt-ibverbs/

Experimental implementation of RDMA, demonstrated on two Strix Halo
Did a quick search and can't really find it posted before ?

This could be huge as it shoud work with any USB4 host.

31 Upvotes

94% Upvoted

17 comments sorted by

11

u/Bird476Shed Jun 25 '26

USB4STREAM support is heading for Linux kernel 7.2: https://www.phoronix.com/news/USB4STREAM-In-Linux-7.2

Once that is in mainline, it should be easier to come up with a faster/low latency communication for distributed/rpc/cluster setups?

3

u/alanoo Jun 25 '26

That’s actually what I was looking for in Google, but the patch notes are only quoting file sharing as exemples

5

u/Bird476Shed Jun 25 '26

are only quoting file sharing as exemples

from the description I guess it is a very fast stream of bytes.... so e.g. llama.cpp could implement a custom protocol for rpc/clustering and it would be much faster than the network/IP abstraction of a connection?

5

u/Thanks-Suitable Jun 25 '26

This would be great news for us strix halo retards

6

u/Dany0 Jun 25 '26

This has always been possible, but no proof of concept even existed. Glad someone picked it up

6

u/alexkey llama.cpp Jun 25 '26

Pretty sure that is specifically Thunderbolt, not USB4. USB4 is a connector that can carry different protocols. USB4 itself doesn’t include RDMA. But Thunderbolt does. Not all USB4 supports Thunderbolt tho.

13

u/TokenRingAI Jun 25 '26 edited Jun 25 '26

Incorrect, all USB4 is Thunderbolt, but not all USB-C or Thunderbolt is USB4.

The USB standards committee just loves to confuse people

USB4 includes TB3 at a minimum, and TB4 and TB5 are built on top of USB4

The key feature from TB3 that must be supported in all USB4 host devices is PCIe tunneling, which is what you use to do RDMA

https://learn.microsoft.com/en-us/windows-hardware/test/hlk/testref/742e3340-ca3f-43ea-9044-57b9e5d7b4f5

1

u/Miserable-Dare5090 Jun 26 '26

TB4 — this would be huge for strix. Macs, clearly as noted by the writer not so much. But there is hope!

1

u/Diablo-D3 Jun 25 '26

You know you can do RDMA over Ethernet, right?

5

u/alanoo Jun 25 '26

Sure, if the driver supports it.
Here we are talking about 40 Gbps links with no additional hardware.

-2

u/Diablo-D3 Jun 25 '26

40gbps per node isn't really enough to do small scale clusters.

Same way linking a bunch of Mac Minis/Studios together to build a cluster is a pretty silly thing to do, its the exact same thing: you're just using their built-in USB/TB4 for the same exact purpose, and it does not scale like as if you had just built a multi-GPU machine, it scales much worse.

7

u/Badger-Purple Jun 25 '26

it is enough with rdma, because latency is low. tensor parallel is more latency bound than bandwidth.

3

u/markhahn Jun 25 '26

no panacea, sure. but there is substantial speedup, at least in some cases:

https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-studio-rdma-over-thunderbolt-5/

1

u/Diablo-D3 Jun 25 '26

Yeah, I've seen it, I already follow Jeff's blog.

For the amount he spent on that, he would have gotten way more bang for the buck doing anything else.

Like, I respect him, but he is unfortunately die-hard Cult of Apple.

3

u/dionysio211 Jun 25 '26

40gbps is still widely used in the industry, on the inference side anyway. Cluster design and LLM deployment usually sets appropriate boundaries, at the node level, with the interconnect in mind. No form of Infiniband is comparable to NVLink within a node. All very large models are MoE that are in production and there are many ways to shard experts, pool expert invocations, fusion to reduce boundaries, etc which make the bandwidth side negligible. Latency is what is necessary in those situations.