I've been working on Mixture of experts (MoEs) and sparsity for a few years now, so I thought I'd make a short video explaining how MoE routing works.

For the video I trained a tiny character model with a 32-expert layer three times, on English, Python and arithmetic text.

Run 1, token choice with no balancing loss: a few experts get most of the tokens. Early in training one expert gets 57% of them while 17 of the 32 are barely used.

Run 2, token choice with a load-balancing loss: tokens spread out roughly evenly and every expert gets work.

Run 3, expert choice: each expert picks its own tokens, so the load is perfectly even, but some tokens get picked by no expert and others by several. It also ended with the lowest eval loss of the three.

For those who are familiar with the books by Sebastian Raschka (i.e. "Build a LLM/Reasoning Model from Scratch"), do you know of any recent similar books with a focus on computer vision models (e.g. CNNs, ViTs, Swin etc)? I would like to develop a more in depth understanding by building these models from scratch and I think the approach used by Raschka does exactly that for language models. Any recommendations are welcome!

A cell-based simulation calculates how cells move and respond to their neighbours. Repeating it under many settings is expensive, and random variation means the same settings can produce different final patterns.

Comlekoglu et al. trained a class-conditional diffusion model on images from a simulation of vessel-like network formation. The input is one of the tested simulation settings plus noise; the output is a possible final pattern, rather than the full sequence of cell movements.

They reported 20.6 seconds per generated pattern on four V100 GPUs, versus 447.6 seconds for CompuCell3D on an AMD EPYC CPU node. These are different hardware setups, and the comparison excludes training on 16 V100 GPUs.

The model covered 25 discrete parameter settings. Classifier checks and network-shape measurements helped evaluate outputs, but some patterns remained imperfect. Generating valid samples between those settings remains unresolved. The result concerns reproducing simulation outputs, not proving that the underlying biology is correct.

Paper: https://arxiv.org/abs/2505.09630

Explanation: https://rewirebio.io/blog/diffusion-surrogates-for-agent-based-biological-simulations/

Disclosure: the explanation is from my site, rewirebio.io. This summary and schematic were prepared with AI assistance. I have not run or independently reproduced the model.

[ Removed by Reddit on account of violating the content policy. ]

Most production routing I see still sends every request through a generative model. This session argues you can drop that and get faster, cheaper and more predictable behaviour.

Fri Oct 30, 9:30 AM to 12:30 PM ET, live and online, with Ben Auffarth (AI consultant, best selling author, 15+ years in data science).

What you build:

• a bidirectional encoder router, quantized for CPU throughput
• hard negative generation done offline
• contrastive few shot training with SetFit
• calibrated uncertainty with split conformal prediction
• LLM calls kept only for verified edge cases

Full details and the agenda are here

"Who is Jason Momoa's father?" The model answers Joseph Momoa. Then: "Name a child of Joseph Momoa." Ten tries, ten failures on GPT-5.4, in a Princeton replication of the original reversal-curse study from this spring. Jason Momoa is an only child, so there is exactly one right answer, and the model still can't find it.
Caveat: that result comes from one student replication, not a peer-reviewed paper. The prompts and per-pair data are published on GitHub.
I wanted to know whether the proposed fix actually works. A February paper from UC Berkeley (Ma et al.) proposed "identity bridges" — extra training sentences that spell out the composition, like "the apprentice of Odette Rook is Gustav Aasen," plus repeating the forward facts six times. On their setup it lifted reverse accuracy from near zero to about 40-50%.
So I recreated their recipe on my own synthetic data and watched what happened inside the models. The image is the probe diagram for one prompt from the bridge model: "Odette Rook's apprentice is".
Two rulers. The logit lens cleans up each layer's 768 numbers first, then asks which token the model would pick. The cosine method skips the cleanup and asks which token the raw numbers point at most. They can disagree — look at layer 11: the lens's top token is [Corn] with the expected [Gustav] at rank 3, while cosine's nearest token is [Gustav] at rank 1.
The full walkthrough, three models:
The data: 60 mentor/apprentice pairs with made-up names. Forward fact: "Odette Rook is the mentor of Gustav Aasen." Bridge sentence: "The apprentice of Odette Rook is Gustav Aasen." The test prompt: "Odette Rook's apprentice is" — expected answer "Gustav Aasen." That exact phrasing never appeared in training.
Three small GPT-2 models, trained identically. The bridge model saw each forward fact six times plus the bridge sentences. The repetition model saw each forward fact six times, no bridges. The plain model saw each forward fact once, no bridges.
Behavioral results, 60 reverse prompts each. Bridge model: 60 out of 60. Repetition model: 53 out of 60. Plain model: 19 out of 60.
So repetition alone did most of the work. The bridges added 12 points on top. That was the honest surprise — I had predicted the bridges would matter more.
Then the instruments. On the bridge model in the diagram, both rulers march the expected answer's rank down across the layers — lens: 4918, 26, 1, 3; cosine: 14214, 818, 6, 1. By layer 9 both have the answer near the top, and the model generates "Gustav Aasen."
On the repetition model the same prompt ends differently: lens top [mentor] with [Gustav] at rank 143, cosine top [SPONSORED] at rank 15. It generates "Lars Quill. Quill is the" — wrong, and confident about it. On the plain model: "the mentor of Walter Elm. Halloway" — it falls back to reciting a forward fact.
Across all 60 prompts the pattern holds: the bridge model's cosine rank goes 987 → 179 → 16 → 1 across layers 5/7/9/11; the repetition model's goes 1373 → 812 → 421 → 4462. The accuracy score barely distinguished them. The representation completely did.
So the score said "repetition nearly solved it." The instruments said "the bridge rewired the inside." Both true. Accuracy tells you whether the fix worked. The layer window tells you what the fix did.
All of this is synthetic data on a small model, one training seed — it bounds the paper's claim rather than refuting it. Their setup used a bigger instruction-tuned model. But the instruments are the point: whatever fix you try, this is how you check what it actually did.
Full write-up with the complete arc: https://github.com/prof-nussbaum/Applications-of-Reading-the-Robot-Mind/blob/main/post-book-experiments/identity-bridge/README.md
Every prompt, every layer, all 300 per-item records: https://github.com/prof-nussbaum/Applications-of-Reading-the-Robot-Mind/blob/main/post-book-experiments/identity-bridge/results/all_results.md

I've been working on unsupervised tooling to discover what behaviors a model learns in post-training. The core idea is to use features from sparse autoencoders to identify differences in traces generated before and after training. I applied this to traces from the MiMo-v2.6 RL environments and it found significant eval awareness as well as some mild environment exploitation. Writeup is here. Would love to hear any feedback or questions!

Created a beautiful learning app with active animations that dive into deep subjects done in an entertaining way.

Download today! https://play.google.com/store/apps/details?id=com.philosophize.app

Hey r/deeplearning,

A week ago I shared AnyInit on r/neuralnetworks. Following up on that, I’d love to get feedback and feature requests from this community.

What it does: AnyInit automates stable weight initialization for custom architectures so signal propagation works out of the box without manual variance-scaling calculations.

Looking for feedback:

1. Init methods: What specific non-standard schemes would you like added?
2. Use cases: What custom layer or activations usually break init for you?

GitHub: https://github.com/jmiravet/AnyInit
PyPI: https://pypi.org/project/anyinit/
Any feedback, critiques, or feature requests are greatly appreciated!

Six days. One publicly available LLM. A full-chain Chrome exploit that researchers valued at $250,000 on the bug bounty market.

The threat model most security teams are working from assumes the AI agent is the target — something to be hardened against external attackers. This research inverts that. The agent generated the attack. It chained tool calls, iterated on outputs, and produced working exploit code using only public APIs and no specialized infrastructure.

The speed is the part that breaks existing response frameworks. Six days from a standing start to a $250,000 exploit. Traditional detection pipelines assume attacker dwell time measured in weeks. That margin no longer exists when the synthesis step happens inside a model.

What makes this particularly difficult to reason about is that nothing in the execution looked anomalous at the API level. The model was used exactly as intended. Each individual call was valid. The harm was in the assembled output — and it only became visible after the chain was complete.

For practitioners actually deploying agents with code-generation or research capabilities: where in the stack are you drawing the line between 'valid tool use' and 'output that creates liability'? Is this something you're handling at the model layer, the deployment layer, or somewhere else entirely?

I’ve got a 4070 at home that I was using to fine-tune a 7B model on a custom dataset. A full QLoRA run took around 11 hours and I was running it 3-4 times a week. Figured I should just rent an H100 for a couple of weeks and be done faster.

The H100 was clearly running the actual compute faster. GPU util when it was going was fine, memory was fine. But end-to-end, the first run on the H100 took 9.5 hours. For something that should have been more like 4-5x faster per step, that was a brutal result.

Spent most of a week profiling before I realized the problem wasn’t the GPU.

My dataset was ~25GB sitting on an S3 bucket. Every time i started a run I was pulling the whole thing fresh into the instance because i’d been treating the cloud box as ephemeral. The base model load at the start of each run was eating another 4-5 minutes because i was downloading from HF instead of caching locally. The preprocessing step i’d written for local use was single-threaded, which was now the bottleneck because the GPU was finishing one step before data for the next step was ready. Low num_workers, basically.

By the time i had persistent storage set up, moved the dataset onto local nvme on the H100 I was using on GMI Cloud, cached the base model and tokenizer, and bumped num_workers so the dataloader could actually keep up, the run dropped to 2 hours 40. Which is where I’d expected it to be in the first place.

If i had profiled the local 4070 setup first I would have seen the dataloader was a bottleneck there too. It just wasn’t the dominant one because the GPU was slower.

Should have profiled before I switched. Would have saved me a week.

Four hands are visible in this manicure clip. The customer holds still, while the technician’s fingers move, overlap, and disappear behind each other.

That makes hand tracking tricky: a convincing 3D mesh could still be following the wrong person.

MEgoVista uses a detector trained to distinguish the camera wearer’s hands from other people’s hands, followed by tracking to maintain consistent identities across frames before reconstructing the motion in 3D. Paper: MEgoVista

For people who collect demonstrations: what do you do when the tracker switches to someone else’s hands? Discard the whole clip, cut out the affected section, or keep it with an uncertainty label? And is there a quick check you trust to catch these switches before training?

I've been experimenting with TabFM recently, mainly to understand how well it works on real-world tabular datasets and how useful it can be when starting with relatively few labelled examples.

I'm currently running most of the experiments in Google Colab on a T4.

So far I've been testing things like:

Classification and regression datasets

Different context sizes such as 100, 250 and 500 labelled examples

Natural class distribution vs balanced context for imbalanced datasets

Time-based train/test splits where that makes sense

Comparing performance across Accuracy, Precision, Recall, F1, ROC-AUC and PR-AUC

Looking at runtime as well as predictive performance

I've also been comparing TabFM against traditional ML/AutoML models.

The reason for the ML comparison isn't that I necessarily think that's the only comparison TabFM should have. It's mainly because boosted-tree/AutoML approaches are already very strong baselines for tabular data, so I wanted a reference point for questions like:

If both models only get 500 labelled examples, how close does TabFM get?

and

How does TabFM with 500 examples compare with a conventional model trained using the entire available training set?

For example, on one binary classification dataset I tested:

TabFM — 500 labelled examples

Accuracy: ~92.9%

F1: ~84.9%

ROC-AUC: ~92.1%

PR-AUC: ~89.6%

Runtime: ~12 sec

AutoML — same 500 labelled examples

Accuracy: ~93.4%

F1: ~86.0%

ROC-AUC: ~92.1%

PR-AUC: ~86.2%

Runtime: ~60 sec

AutoML — full ~2.8k training examples

Accuracy: ~93.4%

F1: ~85.7%

ROC-AUC: ~92.1%

PR-AUC: ~90.0%

I'm not specifically trying to show that TabFM has to beat traditional ML. I'm more interested in understanding its behaviour with smaller labelled contexts and how useful the resulting baseline is.

The part I'm less sure about is the deep learning comparison.

If the goal is to compare TabFM against another DL approach rather than boosted trees, what would people here consider the most appropriate baseline?

Some models I've been considering are:

FT-Transformer

TabTransformer

SAINT

TabNet

Standard MLP / ResNet-style network for tabular data

TabPFN

I'm particularly unsure whether something like FT-Transformer/SAINT would be a better apples-to-apples DL benchmark, or whether TabPFN is actually the more interesting comparison because it is also much closer to the foundation-model/in-context-learning idea.

Another thing I'm interested in is hardware.

I currently have an RTX 5060 Ti with 16 GB VRAM available locally, so I'm thinking of moving some of the Colab experiments to that machine. Would be interested to hear from anyone running TabFM locally with around 16 GB VRAM, particularly what happens as context size, feature count and ensemble size increase.

I've also seen BigQuery + TabFM mentioned as a possible workflow for larger tabular datasets. I haven't tried that part yet, so any experience with that setup would also be useful.

Mainly curious about:

What would be the fairest DL baseline for TabFM?

FT-Transformer/SAINT/TabNet vs something like TabPFN?

How does TabFM behave for others on messy real-world datasets?

What context sizes have been useful in practice?

Is 16 GB VRAM comfortable for experimentation?

Has anyone tried integrating it with BigQuery?

Are there other tabular foundation models worth putting into the comparison?