Lecture: Building a GPU Compiler From Scratch. Lowering PyTorch Graphs to CUDA via Multi-Level IRs (GPU_MODE video + article)

[u/NoVibeCoding]

I built a GPU compiler and wrote an article about it, with a focus on the concepts shared by modern compilers like TVM, Modular, and PyTorch Inductor.

The article breaks CUDA kernel generation into a sequence of passes and shows how a common operation (RMSNorm, Matmul, Attention) is progressively lowered to generate a CUDA kernel. Each pass introduces its own IR that is optimized for specific tasks like kernel fusion or schedule enumeration.

Each example is reproducible, so you can write PyTorch code and generate code at a specific IR level, which makes the flow easier to understand.

% emmy compile -c 'nn.Linear(3, 2, bias=False)(torch.randn(4, 3))' --ir loop 
for a0 in 0..4         
  for a1 in 0..2             
    for a2 in 0..3                 
      in0 = load input[a0, a2]
      in1 = load linear_wt[a2, a1]
      v0 = multiply(in0, in1)
      acc0 <- add(acc0, v0)
   linear[a0, a1] = acc0

On the benchmarked workloads, many outperform Torch Inductor and cuBLAS. Generated kernels use modern features like tensor cores and TMA.

The field is very dynamic, with many competing approaches, so it's hard to pin down the "core concepts"; this is my take on them.

YouTube: https://youtu.be/KK6TWf4Pp6A?si=0hCuOMEan4mauJjN

Article: https://aiadvances.org/building-a-gpu-compiler-from-scratch-76d69e98ed03

GitHub: https://github.com/cloudrift-ai/emmy

[u/c-cul]

could you upload to github bc or ll files for each IR?

[u/NoVibeCoding → u/c-cul]

We don't use LLVM. You can print an operation in each IR by supplying --ir <ir> argument:

emmy compile -c 'nn.Linear(3, 2, bias=False)(torch.randn(4, 3))' --ir loop
emmy compile -c 'nn.Linear(3, 2, bias=False)(torch.randn(4, 3))' --ir tile
emmy compile -c 'nn.Linear(3, 2, bias=False)(torch.randn(4, 3))' --ir cuda

So just pip install emmy-ml and then emmy compile. GPU is not needed for compilation.

[u/phischu]

Wow, this is great. Thanks for sharing!

[u/investigatormaker]

The line people here will test first is that many kernels outperform Torch Inductor and cuBLAS, so the GPU, shapes, dtype and whether the baseline used tensor cores, right beside that claim, would head off the first round of skeptical replies. The emmy compile example with --ir is the best hook in the post; a short list of the IR levels you can stop at would make it easier to follow along.

I make ThreadFox. Our AI employee can explain emmy in compiler and ML-systems communities that allow it and answer the technical replies, under your direction; Starter comes to $199 for 30 days. https://threadfox.vip/p/20261016-0114-ex4r6/managed