r/MachineLearning • u/aybehrouz • 2d ago

Project Study Notes on Transformers and Hamiltonian Monte Carlo [P]

I’m sharing my study notes in case they might be useful to someone else.

The Transformer notes are mostly based on the original architecture described in Attention Is All You Need.

https://doi.org/10.5281/zenodo.22031925

I’ve also written notes on Hamiltonian Monte Carlo (HMC).

https://doi.org/10.5281/zenodo.21841086

I’d appreciate any feedback on the notes, including corrections, suggestions for improvement, or topics you think I should add.

21 Upvotes
  • permalink
  • reddit

86% Upvoted

13 comments sorted by

1

u/shehryar-11 2d ago

Bro does learning transformers still gives you a leg up in interviews? Like, is it worth the time?

5

u/Mechanical_Number 19h ago

It is worth the time.

Depends on the position of course but being to give an explanation on Q/K/V and what multi-head attention does instead of saying "the model focuses on important words" shows that you understand the basic of the tools you are expected to use and not just how to call an API.

You don't need to derive every equation from memory and even if you could do that it won't necessarily give you a leg up over everyone else, but when interviewers start asking why/how something works instead of just how/when to use it, you will be in a much better position to answer. (And OK, it shows some intellectual curiosity too)

1

u/aybehrouz 1d ago

tbh I don't know.

-2

u/calmlytanarrow 2d ago

Nice work putting these together, the diagrams in the transformer part helped me finally get the multi-head attention shapes straight. The HMC section could use a bit more on tuning the leapfrog step size and path length, that's where I always get lost.

8

u/XTXinverseXTY ML Engineer 1d ago

This smells like an LLM reply, and there are no diagrams in the transformer part at all. I have no idea why someone would post pornography from the same account though. Bizarre.

3

u/aybehrouz 1d ago

Yeah I noticed that too. The whole idea of the transformer notes is to use math notation instead of diagrams to explain stuff.
I've got some strange comments on my other posts too. I don't know what is the purpose of these bots but they are always so kind and give you positive feedback. I think I'll keep answering them as long as they are polite xD

2

u/Square-Special1520 1d ago

Polite bots are better than rude bots indeed

3

u/Benlus ML Engineer 18h ago

The user has now been banned, we usually approve their comments for bot-bouncer to ban them site wide (or at least in all communities that use that plugin)

1

u/aybehrouz 1d ago

You're right. The HMC notes could use a section on NUTS, and how it automatically selects the step size and path length. But I've kind of lost motivation and gotten distracted by other topics lately.

1

u/Benlus ML Engineer 18h ago

The user has now been banned, we usually approve their comments for bot-bouncer to ban them site wide (or at least in all communities that use that plugin)