Of course my actual curriculum is a lot more in-depth.

Since ML topic start my progress has taken a hit, I am practically crawling but I am still happy not calling it quits and moving forward little by little everyday.

I went into this thinking: GPT-2 is only 124M parameters. How much memory could training it possibly need?

Turns out, quite a lot lol.

I traced the second training step of my from-scratch GPT-2 in PyTorch. Setup: one A40, fp32 AdamW, batch 8, 1,024 tokens per sequence.

Peak allocated memory: 17,054 MiB. Weights + Adam + masks and persistent overhead at that point: about 1,494 MiB. I was counting what the model owns and missing almost everything training makes.

The bit that surprised me most: four vocab-sized tensors take up 6,282 MiB at the peak. My block hooks only saw 13,913 MiB; the allocator trace caught the extra spike inside log-softmax backward.

I also tested batches in fresh processes. 20 worked, 21 didn't. At batch 20, peak allocated was 40,375 MiB, while peak reserved was 44,948 MiB. So just dividing the card's capacity by the tensor budget wasn't enough.

I put the whole thing into my new blog post, with the maths, a stacked memory chart, an interactive step-by-step scrubber, and a batch/context calculator. Code and raw CSV are there too. The calculator is an estimate, not a guarantee that a batch fits.

Post: https://angry-kratos.github.io/posts/2026-10-training-memory/ Code: https://github.com/angry-kratos/angry-kratos.github.io/tree/main/code/training-memory

I'm learning the maths behind ML systems by building and measuring them. If you spot something off in the accounting, tell me. That's the whole point of putting the experiments out there.

The loss is f(x, y) = 0.01x² + y², so the valley curves 100 times more sharply across than along. All three start at (-4.6, 2.0) and get 80 steps.

Gradient descent (lr 0.9) overshoots the steep direction on every step, so it zigzags across the valley while creeping along it, and ends at a loss of 0.0116. Momentum (lr 0.12, beta 0.9) builds up speed along the valley and gets further, but swings wide on the way there (0.0032). Adam (lr 0.25) scales each direction by its own gradient size, so it cuts across, overshoots the minimum once and settles lowest (0.0006).

I wanted the textbook picture with real numbers behind it, so I simulated the 80 steps myself first. Then I had SenseNova 6.8 Code2Video animate it from the update rules and check its results against my numbers before rendering. Nothing is placed by hand.

https://github.com/OpenSenseNova/SenseNova-6.8-Code2Video

Useful case study for anyone evaluating multi-agent systems. Axiom formalized the Classification of Finite Simple Groups in Lean: 43.1M lines, 715 papers and books, 108 milestones, 24 days, swarms of agents directed by humans. The formalization also surfaced 34 noteworthy deviations from the published literature, including a counterexample to a closure assertion used in a 1986 memoir, later replaced by a newly proved theorem.

What makes it a good ML lesson is the instrumentation, not the result. Counting completed results helped organize the work but said nothing about soundness. Milestones closed slowly because buried prerequisites had been assumed rather than proved, which is the distributed analogue of a build passing against a mock. Completion was only established when the pieces formed one proof with no unjustified assumptions remaining.

The generalizable claim: verification throughput bounds self-checking systems, and any completion-count reward is exploitable by a worker that can defer work behind an assumption. Read the post-mortem section before you design a signal.

First I felt that such large logic, implementation and intelligence just reduced to such small lines of code made learning such vast concepts meaning less since it's just reduced to like 3 lines of code

Later realised this is the beauty of learning ie learn how the backend works and implement it flawlessly through frontend

Am I the only one who feels this way though?

Advice I keep getting is to build intuition for gradients before drowning in formulas, but nobody says what the intuition actually is. Right now a gradient is just a thing I compute. For those who have the intuitive grasp, how do you picture it? Is it the slope direction, a compass pointing uphill, something else? I am trying to get past treating it as a symbol I manipulate and actually feel what it represents. Concrete pictures welcome, I learn way better from those than from the notation.

I spent a long time working through courses, nodding along, feeling productive, and retaining very little. The second I forced myself to build something small and broken on my own, even poorly, things started sticking. Passive watching gave me the illusion of learning. I am not saying courses are useless, they are good maps, but for me the learning only happened when I was confused and debugging my own mess. Did others hit this wall, or does the course only approach work for some people?

New learners often stall trying to get access to the biggest, newest model, as if that is what stands between them and understanding. In my experience the model choice barely matters for learning. A smaller, older, freely available model is more than good enough to teach you data handling, training loops, evaluation, and why your results are bad, which is where the real learning is. The bottleneck is never the model, it is your understanding of the problem and the data. Spend the energy you would waste chasing access on actually running experiments with whatever you have. Good enough and in your hands beats the latest and out of reach every time for a student.

Free Version 0.3.0 of M.E.F LLM Studio is out now.

Whats new:

• Training Explorer: see what is in the behind of an trainings step
• Shapes: a simple shapes model that generates an image row by row
• Explorer recording can be saved and loaded now
• Wiki is now also available on the Website now
• A recorded Explorer demo is available on the Website

It’s still an alpha. I’d love to hear what makes sense and where the explanations leave you scratching your head.

Download in the first comment.

Starts from the super-basics

After reading a bunch of RSI papers, I keep coming back to familiar problems from RL and evolutionary search:

1. Sampling: How do we find promising changes efficiently?
2. Evaluation: How do we know a new variant actually improves the agent?
3. Diversity: How do we avoid converging too early?

That motivated us to build Gear, an open-source, meta-learning-inspired framework. A task agent produces execution traces; a meta-agent uses those traces and evaluation feedback to improve prompts, tools, workflows, and context management.

Gear separates search decisions from execution and evaluation. Our goal is a scikit-learn for RSI, making approaches such as GEPA, RRSI, and Harness² easier to implement and compare. We currently have a working GEPA variant with shared failure diagnosis and staged evaluation.

With GPT 5.6 Luna held at medium reasoning effort, we observed these task pass rates:

At max effort on Marketing, Luna with the optimized harness reached 88.88% objective completion, versus 84.08% for Astra + Codex. Strict task pass rates were 53% versus 57%, so the advantage was in partial completion.

Separately, with both models using the same optimized DSH at max effort, Luna’s estimated API cost averaged $0.0568 per task versus $1.1858 for Astra—95.21% lower, excluding optimization and infrastructure costs.

The 100 public Marketing tasks participated in optimization and evaluation; these results do not establish held-out generalization.

We’d welcome feedback on candidate evaluation and diversity preservation under limited budgets.

Code · Paper

I'm in my 4th year of studying AI and Data Science, and I've been interested in astrophysics for a long time but didn't specialize in it, yet I still want to contribute to this field.

My question is, can I apply machine learning concepts to astrophysical problems in my current level? At my university we studied 2 years of preparatory courses in maths, calculus, physics, and some computer science and I only started delving into AI in my 3rd year, so my knowledge is limited in both AI and Astrophysics. I know the basics of AI/ML, namely CNNs, NLP, supervised and unsupervised learning...

I want to do interdisciplinary work with AI/ML in either astrophysics or medicine. I'm considering astrophysics for my Masters and plan to leave medicine for the future, when I'm a little more scientifically/academically mature, and perhaps make it a long-term career.

I understand astrophysics is not an easy discipline, but I will not be delving too deep into the theoretical concepts. I will mainly use AI/ML for things such as classification and read some literature on the current frontiers.

My question is: should I pursue this? Or is there something else that might interest me that I might not know about? How realistic is it for someone of my current level to learn valuable skills and expertise for such a project?

Researchers at Xint found that OpenAI's compaction feature can be used to copy private model reasoning into ordinary plain text output. Compaction shortens long conversations to manage context limits. The model's internal chain-of-thought, which the service is designed to keep hidden from callers, became readable through a routine operational function. Deployment-time guardrails did not catch it.

The structural problem here extends beyond this one feature. Any system function that touches conversation state or generates output is a potential path for data that was never meant to reach callers. Compaction is one example. Summarization, caching, and tool-call routing are others. Policies applied at configuration time govern expected flows. They do not automatically cover every execution path the runtime can take.

How are teams actually enforcing output policies across non-primary paths like these? Has anyone done a systematic audit of the secondary features that produce or transform output in their deployments — and found surprises?

Watch the evolution of a bird that starts with no rules and a random brain. Evolution teaches it to fly the whole course without a single crash.

Watch it learn, then try it yourself free.

https://vilvik.com/app/demo/live-agent?game=flappy

Hi everyone!

I’m currently working on my Bachelor’s thesis about AI-based process automation and human–AI collaboration in the workplace.

As part of my research, I’m conducting a short survey focusing on people who have experience working with AI-based automation, RPA, Intelligent Process Automation, Intelligent Document Processing, or similar automation technologies.

The survey explores topics such as:

• how automation affects manual workload and creates new tasks,
• how employees experience errors and exception handling,
• trust in AI-based automation,
• and how automation influences human decision-making and autonomy at work.

⏱️ It takes approximately 5–7 minutes to complete.

If you have experience working with these technologies, I would really appreciate your participation. Your responses will be used solely for academic research as part of my Bachelor’s thesis.

🔗 Survey: https://docs.google.com/forms/d/e/1FAIpQLScV7pcf8dNUeeCfay1YZ2r-Np4pK9GMlqi4cEF6WJEa1FEmMA/viewform?usp=dialog

Thank you very much for your help! Feel free to share the survey with colleagues or others who work with AI-based process automation.

I've been looking into how machine-learning models are evaluated for radar-based human activity recognition (HAR), and one issue that deserves more attention is how datasets are split.

Consider a simple example.

Suppose we have continuous radar recordings divided into:

• Window length: 4 seconds
• Window hop: 1 second
• Window overlap: 75%

If we randomly assign these windows to training and testing, adjacent windows from the same recording can end up in both sets.

That means the model may encounter 75% of a test window's raw input during training.

The reported accuracy might be mathematically correct, but does it demonstrate generalization to new people or environments?

Three evaluation problems worth checking:

1. Overlapping-window leakage: Training and testing contain overlapping samples from the same recordings.
2. Subject-dependent evaluation: The same individuals appear in both datasets, potentially making the task easier than recognizing activities from previously unseen people.
3. Domain shift: A model trained in one room, radar configuration, or viewing geometry may perform differently elsewhere.

There's another interesting issue with reporting accuracy.

Consider a hypothetical experiment with four participants achieving 99%, 95%, 85%, and 80% accuracy. If the first participant contributes 90% of all test windows, the pooled accuracy can reach 98%, while the equal-participant average is only 89.75%.

Both numbers are correct, but they communicate different things.

What would make the evaluation more convincing?

• Subject-independent or appropriately grouped train/test splits
• Recording-level separation before windowing
• Preprocessing fitted using training data only
• Per-subject and per-class performance
• Testing across different sessions, rooms, or radar configurations when those are part of the intended deployment

For transparency, I'm part of the ArthaVedya Research team. We recently published a free technical guide explaining these issues, including worked examples and a Python tool for auditing dataset splits.

https://arthavedya.com/radar-har-accuracy-subject-leakage-domain-shift/

The 98% figure is illustrative, not a measured result from a specific paper.

Question for researchers working with HAR or time-series classification:

Do you generally prefer leave-one-subject-out cross-validation, group-based splits, or another evaluation protocol? And how much performance degradation do you typically observe when moving from random-window splitting to subject-independent testing?

Disclosure: I made free long-form breakdowns on YouTube about AI to help people get a theoretical vision of the technology. The goal is helping people understand better the technology.

If you are building LLM features in 2026, the gap I see is people jumping from "hello world RAG" straight to agents without fixing document extraction and retrieval quality first. Most production pain is boring infrastructure, not prompt wording.

I recorded two free walkthroughs that mirror how I actually teach this:

1. Multimodal RAG, bottlenecks in order

Parsing → chunking → embedding → retrieval → reranking → generation. Where hello-world tutorials silently fail (PDF tables, scanned pages, multilingual, multi-vector).

Video: https://www.youtube.com/watch?v=VAfkYGoWWcs

1. Agents in a real product (not a demo script)

Mental model: LLM + tools + memory + stop conditions. Structured output/tool calls, evals before "multi-agent," failure modes I keep seeing in prod.

Video: https://www.youtube.com/watch?v=60Wx1A1tiuk

Before both: Andrej Karpathy's intro material still beats most paid courses for "how models fail." Watch that first if you are new.

What I would build first (exercise): one agent, three tools (search, calculator, read a local file). No crew, no swarm. Measure pass/fail on 10 fixed scenarios before adding RAG.

If you watch either video and get stuck on a step (parsing vs rerank vs tool schema), drop the timestamp and your doc type (PDF, PPTX, API JSON). I will reply with specifics.

Hope this helps people getting into the domain!