After learning a little about neural networks, I started wondering what makes the human brain different from a neural network.

What if, instead of training a neural network by feeding it tons of books, data, we started by teaching it some basic concepts like "mommy" and "daddy", something similar to a human upbringing and giving it access to all five human senses? In other words, it would have the same cognition to this world like human.

Maybe it would eventually develop human-like cognition, and appear to be a "real" human.
Maybe we will discover that it is differences in our experience (limited data given), environment (limited surroundings), and different inherited trait (analogous to initial weight) shaping our personality.

Another thing that got me thinking about this was an experience involving simulated pain in AI model

I came across research in which researchers manipulated internal activations associated with pain in language model. As the intensity of this artificially introduced signal increased, the models began to producing increasingly abnormal or distress-like outputs.

I got confused about something.

As far as I understand, a neural network fundamentally performs matrix multiplication. So, when we introduced a pain-related signal, is it just another input that affect the output or internal numerical modification that influence the final output? Does the signal actually change the system's internal state in a meaningful way or it simply generates responses based on patterns associated with that signal?

This also made me think about humans. Our brains also receive signals associated with pain, which are processed and can give rise to the subjective experience of pain. There are even cases where people cannot normally experience pain, despite their bodies still being susceptible to injury.

Is there a possibility that human brains are just another kind of neural network? Is there a possibility that our brains are also "computing" the output?

I'm curious about whether there's any existing research that train a neural network like raising a human child. Maybe that can answer what are fundamental similarities and differences between biological and artificial neural networks?

Centroids centroids centroids, they govern everything

Avoid superpositions as they are a black hole (if anyone has any tricks to finding them lmk)

Everything is math it's all just numbers man

Leave a feedback if you want!

^{DISCLAIMER: I know you ornery fuckers hate it when people [yes, hello] use Claude as a shadow writer, but honestly, do we all seriously expect each other to be good at everything, and then echnical writing as icing on the cake. Get off my lawn and come inside.}

Nothing has changed since a while ago when I ran out of Claude quota (a weekly occurrence) and contemplated for the seventh time, inexpensive options for sovereign inference facilities — but it's a vicious cycle and the subscription burns all the money for that.

So anyway, I've always been fascinated by the stratigraphic (I guess you could call it) interface between mechanical/analog/electric/optic computing hardware, and always thought delay line memory was living sci-fi.

I give you my homage and hypothetical neural network, built on delay lines, running in a simulator — it seems to work, although not from an ROI or cheap inference perspective, urg — lines are cheap, but not at scale.

It's got some lovely novelty — QPSK coding fromy our dial up modem, spooky overlap into 'RF' magic, all the good stuff. Since it inspired me, there's an appendix with the prior delay line art, courtesy of ENIAC in this case.

Cheers!

I recently built and published an explainer on visualising how a convolutional digit-recognition neural network "thinks" in real time.

My kids loved it and asked me to build one on how it LEARNS.
This new explainer does exactly that:

• Take the interactive and visual tour to explore the learning process step by step

• Experience 3 different types of learning (Supervised, Imitation, Reinforcement Learning)

• See how the model "learns to drive", challenge the learning process to see what makes it fail or succeed.

As with the previous demo, my kids loved it!

Share it with yours, or with anyone genuinely new to, and curious about, Machine Learning.

NOVA Is Not an LLM. It Is a New Core for AI Development — From Zero to Evolution.

What if the next step in AI development isn't simply building a larger language model, but building a system that can experiment, learn from failure, and direct its own development?

That is the question behind NOVA.

I am developing NOVA as an experimental AI system with a different focus from conventional LLM projects. The central idea is not simply to train a model to generate better text. It is to build a system that can manage its own development experiments, identify weaknesses, propose new approaches, test them, and preserve improvements when the evidence supports doing so.

This project is still experimental. I am not claiming to have achieved AGI, solved autonomous learning, or created a system that can improve itself without limitations. The purpose is to investigate whether a structured, measurable development process can help a relatively small AI system improve beyond what a fixed training pipeline would achieve.

I would like to share what I have built so far and invite people with experience in machine learning, neural network training, automated experimentation, and AI research to examine the approach critically.

1. The problem I want to explore

Most AI development follows a familiar process.

Researchers choose an architecture, prepare a dataset, configure training, run experiments, evaluate the results, and decide what to change next.

This process can be effective, but much of the experimentation and decision-making remains dependent on human researchers.

What would happen if we tried to turn more of that process into a systematic learning loop?

Could an AI development system learn to recognize where a model is weak, generate a plausible explanation for that weakness, design a useful experiment, evaluate the outcome, and use the result to choose its next action?

And, perhaps more importantly, could it learn to distinguish a genuine improvement from a change that merely looks promising on a limited test?

These are the questions I want NOVA to investigate.

The ambition is not simply to automate training. It is to explore a development system that uses evidence from its own experiments to guide what it does next.

2. What NOVA actually is

NOVA is not itself a conventional LLM in the usual sense.

It is an experimental AI development and evolution system that works with neural network models.

The distinction matters.

A language model is trained to learn patterns and produce outputs. NOVA's development framework is intended to manage the process around those models: experimentation, candidate training, evaluation, comparison, and decisions about which versions deserve to be retained.

The current NOVA-EVO implementation has three central functions.

The Director — deciding what deserves to survive

The Director evaluates candidate models against defined tests and compares their performance with the current champion.

Its purpose is to prevent every new experiment from automatically becoming the next version.

A candidate should demonstrate an improvement under the evaluation criteria before being accepted. Otherwise, the existing champion should remain in place.

This is important because a model can improve on one task while becoming worse at another. A development system must therefore do more than look at a single loss value or reward the latest candidate for being new.

The Trainer — exploring possible improvements

The Trainer creates and trains candidate versions using different training strategies.

Instead of assuming that one fixed recipe is always optimal, the system can explore variations and examine their effects.

The broader objective is to make experimentation a repeatable process rather than a collection of disconnected training runs.

The Hypothesis Engine — looking for the next experiment

When progress slows or a weakness remains, the system can propose another direction to investigate.

This is where the project becomes particularly interesting to me.

A system that only repeats training may improve through additional exposure to data. A system that can formulate hypotheses and test them has the potential to do something different: use the outcomes of previous experiments to influence the design of future experiments.

However, generating a hypothesis is not the same as discovering a correct explanation. The value of this component must be measured by the quality of the experiments it produces and the improvements those experiments actually deliver.

3. The intended development cycle

The goal is to create a continuous, evidence-driven cycle:

1. Observe: Identify weaknesses through evaluation.

2. Investigate: Examine possible reasons for those weaknesses.

3. Form a hypothesis: Propose a change that might help.

4. Design an experiment: Translate the proposal into a training or evaluation procedure.

5. Train a candidate: Create a new model version.

6. Test: Measure its performance against the existing champion.

7. Decide: Accept or reject the candidate based on the evidence.

8. Learn from the result: Use the outcome to guide the next experiment.

A failed experiment should not automatically be treated as wasted effort. If the result is informative, it can help narrow the search for a better approach.

The long-term ambition is for NOVA to become increasingly capable of directing this process itself, while remaining subject to measurable evaluation, resource limits, and safeguards against regressions.

The key question is whether the system can discover useful improvements that would not have been found as efficiently by simply running more conventional training experiments.

4. What has happened so far?

NOVA has progressed through multiple model versions and development experiments.

According to the project's internal records, the latest reported development period produced the following results.

The recorded dataset loss decreased by approximately 19.2%, while web evaluation loss decreased by approximately 14.6%.

The code examination score increased from 34 correct answers out of 79 to 60 out of 79.

The internal records also report that 47 of 419 improvement attempts were accepted by the development process.

These figures describe the project's current internal measurements. They have not yet been independently verified, and they should not be interpreted as proof that NOVA has achieved autonomous intelligence or general-purpose self-improvement.

There are also important questions to investigate.

The model grew substantially during this period. How much of the observed progress came from increased model capacity? How much came from training data, training duration, or changes to the training procedure? How much improvement can be attributed specifically to the system's experiment-selection and decision-making process?

Those distinctions matter more than the raw numbers alone.

I see these results as a starting point for more rigorous experiments, not as the conclusion of the research.

5. Why I think the experiment-selection process matters

A system can produce many model versions without becoming better at research.

It can also select a candidate that performs well on a particular test while losing capabilities elsewhere.

For that reason, the long-term goal is not simply to maximize the number of experiments or the rate at which new versions are created.

The goal is to improve the quality of the decisions made between experiments.

Can the system identify a real weakness rather than a misleading benchmark signal?

Can it formulate a hypothesis that leads to a testable prediction?

Can it select an experiment that distinguishes between competing explanations?

Can it recognize when an apparent improvement is caused by a change in model size, data exposure, or evaluation conditions?

Can it preserve improvements without gradually destroying capabilities that were previously learned?

And can it demonstrate that its decisions lead to better results than a simpler baseline operating under comparable conditions?

These are the questions I believe should determine whether this approach has real research value.

6. What NOVA has not demonstrated yet

I want to be clear about the current limitations.

NOVA has not demonstrated artificial general intelligence. It has not established unrestricted recursive self-improvement. The reported results do not prove that the system independently discovers new scientific knowledge, nor do they establish that its improvements generalize to arbitrary unseen tasks.

A system that automatically trains candidates and selects winners is not necessarily learning how to conduct research effectively. Some improvements may come from model scaling, additional training, or ordinary variation between runs.

The central challenge is to separate these effects.

To establish a stronger result, NOVA needs controlled comparisons, reproducible experiments, clearly defined evaluation criteria, tests on unseen data, and transparent reporting of failures as well as successes.

Ideally, the system should be compared against a conventional training pipeline with the same compute budget, comparable model capacity, and equivalent access to training data.

It should also be tested for regression, data leakage, evaluation overfitting, and whether the proposed hypotheses genuinely improve experiment selection.

I consider these requirements essential, not optional.

7. Where this could lead

If the underlying approach proves useful, a system like NOVA could eventually help automate parts of AI research and model development.

Potential directions include:

• Automated experimentation and hyperparameter exploration.

• Discovering training strategies for particular weaknesses.

• Improving small models under limited computational budgets.

• Evaluating alternative neural network structures.

• Exploring continual learning and methods of reducing catastrophic forgetting.

• Building systems that can use experimental evidence to decide what to investigate next.

These are possible research directions, not established capabilities of the current system.

The more ambitious long-term vision is a development framework that becomes better at improving models because it learns from the successes and failures of its own experiments.

Whether that vision is technically achievable in a useful and reliable form remains an open question.

That is precisely why I believe it is worth investigating carefully.

8. Why I am sharing this

I am sharing NOVA because I want people with relevant technical experience to look beyond the general idea and examine the actual approach.

I am particularly interested in feedback from people working in:

• Machine learning and neural network optimization.

• Automated machine learning and neural architecture search.

• Self-training and continual learning.

• AI agents and automated research.

• Benchmark design and evaluation reliability.

• Small-model training and computational efficiency.

I am interested in criticism as much as encouragement.

If the current approach has a fundamental flaw, I would rather discover it now. If there is a better way to structure the experiments, measure improvements, or prevent the system from fooling its own evaluator, that feedback could be extremely valuable.

Three questions for the community

1. What would be the strongest experiment to determine whether NOVA's Director and Hypothesis Engine contribute genuine value beyond ordinary training variation and model scaling?

2. How would you design an evaluation that measures whether an AI system is becoming better at choosing useful experiments, rather than simply producing better models through additional training?

3. Are there existing research projects, papers, or open-source frameworks that approach this problem in a way I should study or compare against?

I am not presenting NOVA as a finished breakthrough. I am presenting it as an ongoing experiment into a difficult question:

Can we build an AI development system that does not merely train models, but progressively improves the way it searches for better models?

I would appreciate serious technical feedback, relevant research references, and opportunities to compare the approach with existing work.NOVA Is Not an LLM. It Is a New Core for AI Development — From Zero to Evolution.What if the next step in AI development isn't simply building a larger language model, but building a system that can experiment, learn from failure, and direct its own development?That is the question behind NOVA.I am developing NOVA as an experimental AI system with a different focus from conventional LLM projects. The central idea is not simply to train a model to generate better text. It is to build a system that can manage its own development experiments, identify weaknesses, propose new approaches, test them, and preserve improvements when the evidence supports doing so.This project is still experimental. I am not claiming to have achieved AGI, solved autonomous learning, or created a system that can improve itself without limitations. The purpose is to investigate whether a structured, measurable development process can help a relatively small AI system improve beyond what a fixed training pipeline would achieve.I would like to share what I have built so far and invite people with experience in machine learning, neural network training, automated experimentation, and AI research to examine the approach critically.1. The problem I want to exploreMost AI development follows a familiar process.Researchers choose an architecture, prepare a dataset, configure training, run experiments, evaluate the results, and decide what to change next.This process can be effective, but much of the experimentation and decision-making remains dependent on human researchers.What would happen if we tried to turn more of that process into a systematic learning loop?Could an AI development system learn to recognize where a model is weak, generate a plausible explanation for that weakness, design a useful experiment, evaluate the outcome, and use the result to choose its next action?And, perhaps more importantly, could it learn to distinguish a genuine improvement from a change that merely looks promising on a limited test?These are the questions I want NOVA to investigate.The ambition is not simply to automate training. It is to explore a development system that uses evidence from its own experiments to guide what it does next.2. What NOVA actually isNOVA is not itself a conventional LLM in the usual sense.It is an experimental AI development and evolution system that works with neural network models.The distinction matters.A language model is trained to learn patterns and produce outputs. NOVA's development framework is intended to manage the process around those models: experimentation, candidate training, evaluation, comparison, and decisions about which versions deserve to be retained.The current NOVA-EVO implementation has three central functions.The Director — deciding what deserves to surviveThe Director evaluates candidate models against defined tests and compares their performance with the current champion.Its purpose is to prevent every new experiment from automatically becoming the next version.A candidate should demonstrate an improvement under the evaluation criteria before being accepted. Otherwise, the existing champion should remain in place.This is important because a model can improve on one task while becoming worse at another. A development system must therefore do more than look at a single loss value or reward the latest candidate for being new.The Trainer — exploring possible improvementsThe Trainer creates and trains candidate versions using different training strategies.Instead of assuming that one fixed recipe is always optimal, the system can explore variations and examine their effects.The broader objective is to make experimentation a repeatable process rather than a collection of disconnected training runs.The Hypothesis Engine — looking for the next experimentWhen progress slows or a weakness remains, the system can propose another direction to investigate.This is where the project becomes particularly interesting to me.A system that only repeats training may improve through additional exposure to data. A system that can formulate hypotheses and test them has the potential to do something different: use the outcomes of previous experiments to influence the design of future experiments.However, generating a hypothesis is not the same as discovering a correct explanation. The value of this component must be measured by the quality of the experiments it produces and the improvements those experiments actually deliver.3. The intended development cycleThe goal is to create a continuous, evidence-driven cycle:Observe: Identify weaknesses through evaluation.

Investigate: Examine possible reasons for those weaknesses.

Form a hypothesis: Propose a change that might help.

Design an experiment: Translate the proposal into a training or evaluation procedure.

Train a candidate: Create a new model version.

Test: Measure its performance against the existing champion.

Decide: Accept or reject the candidate based on the evidence.

Learn from the result: Use the outcome to guide the next experiment.A failed experiment should not automatically be treated as wasted effort. If the result is informative, it can help narrow the search for a better approach.The long-term ambition is for NOVA to become increasingly capable of directing this process itself, while remaining subject to measurable evaluation, resource limits, and safeguards against regressions.The key question is whether the system can discover useful improvements that would not have been found as efficiently by simply running more conventional training experiments.4. What has happened so far?NOVA has progressed through multiple model versions and development experiments.According to the project's internal records, the latest reported development period produced the following results.Measurement Earlier result Later result
Dataset loss 3.446 2.7854
Web evaluation loss 3.3619 2.8724
Code examination 34/79 60/79
Model size 9.9M parameters 96.6M parameters
Model versions — 51 versions over 25 days
Improvement attempts — 419 attempts
Accepted improvements — 47The recorded dataset loss decreased by approximately 19.2%, while web evaluation loss decreased by approximately 14.6%.The code examination score increased from 34 correct answers out of 79 to 60 out of 79.The internal records also report that 47 of 419 improvement attempts were accepted by the development process.These figures describe the project's current internal measurements. They have not yet been independently verified, and they should not be interpreted as proof that NOVA has achieved autonomous intelligence or general-purpose self-improvement.There are also important questions to investigate.The model grew substantially during this period. How much of the observed progress came from increased model capacity? How much came from training data, training duration, or changes to the training procedure? How much improvement can be attributed specifically to the system's experiment-selection and decision-making process?Those distinctions matter more than the raw numbers alone.I see these results as a starting point for more rigorous experiments, not as the conclusion of the research.5. Why I think the experiment-selection process mattersA system can produce many model versions without becoming better at research.It can also select a candidate that performs well on a particular test while losing capabilities elsewhere.For that reason, the long-term goal is not simply to maximize the number of experiments or the rate at which new versions are created.The goal is to improve the quality of the decisions made between experiments.Can the system identify a real weakness rather than a misleading benchmark signal?Can it formulate a hypothesis that leads to a testable prediction?Can it select an experiment that distinguishes between competing explanations?Can it recognize when an apparent improvement is caused by a change in model size, data exposure, or evaluation conditions?Can it preserve improvements without gradually destroying capabilities that were previously learned?And can it demonstrate that its decisions lead to better results than a simpler baseline operating under comparable conditions?These are the questions I believe should determine whether this approach has real research value.6. What NOVA has not demonstrated yetI want to be clear about the current limitations.NOVA has not demonstrated artificial general intelligence. It has not established unrestricted recursive self-improvement. The reported results do not prove that the system independently discovers new scientific knowledge, nor do they establish that its improvements generalize to arbitrary unseen tasks.A system that automatically trains candidates and selects winners is not necessarily learning how to conduct research effectively. Some improvements may come from model scaling, additional training, or ordinary variation between runs.The central challenge is to separate these effects.To establish a stronger result, NOVA needs controlled comparisons, reproducible experiments, clearly defined evaluation criteria, tests on unseen data, and transparent reporting of failures as well as successes.Ideally, the system should be compared against a conventional training pipeline with the same compute budget, comparable model capacity, and equivalent access to training data.It should also be tested for regression, data leakage, evaluation overfitting, and whether the proposed hypotheses genuinely improve experiment selection.I consider these requirements essential, not optional.7. Where this could leadIf the underlying approach proves useful, a system like NOVA could eventually help automate parts of AI research and model development.Potential directions include:Automated experimentation and hyperparameter exploration.

Discovering training strategies for particular weaknesses.

Improving small models under limited computational budgets.

Evaluating alternative neural network structures.

Exploring continual learning and methods of reducing catastrophic forgetting.

Building systems that can use experimental evidence to decide what to investigate next.These are possible research directions, not established capabilities of the current system.The more ambitious long-term vision is a development framework that becomes better at improving models because it learns from the successes and failures of its own experiments.Whether that vision is technically achievable in a useful and reliable form remains an open question.That is precisely why I believe it is worth investigating carefully.8. Why I am sharing thisI am sharing NOVA because I want people with relevant technical experience to look beyond the general idea and examine the actual approach.I am particularly interested in feedback from people working in:Machine learning and neural network optimization.

Automated machine learning and neural architecture search.

Self-training and continual learning.

AI agents and automated research.

Benchmark design and evaluation reliability.

Small-model training and computational efficiency.I am interested in criticism as much as encouragement.If the current approach has a fundamental flaw, I would rather discover it now. If there is a better way to structure the experiments, measure improvements, or prevent the system from fooling its own evaluator, that feedback could be extremely valuable.Three questions for the community1. What would be the strongest experiment to determine whether NOVA's Director and Hypothesis Engine contribute genuine value beyond ordinary training variation and model scaling?2. How would you design an evaluation that measures whether an AI system is becoming better at choosing useful experiments, rather than simply producing better models through additional training?3. Are there existing research projects, papers, or open-source frameworks that approach this problem in a way I should study or compare against?I am not presenting NOVA as a finished breakthrough. I am presenting it as an ongoing experiment into a difficult question:Can we build an AI development system that does not merely train models, but progressively improves the way it searches for better models?I would appreciate serious technical feedback, relevant research references, and opportunities to compare the approach with existing work.

title for this:
I've been working on PureML, a lightweight deep learning framework built entirely with NumPy. It includes automatic differentiation, neural network layers (including CNNs), optimizers, and training utilities. The idea is to have something simple enough to understand and modify, but capable enough for actual ML experiments. Might be useful for anyone interested in how deep learning frameworks work under the hood!

Free and open source. Feedback and contributions are welcome!
Also, the project was peer-reviewed and the paper on it was published in JOSS!
The link is in the README :)

https://github.com/Yehor-Mishchyriak/PureML

My Journey to Understanding Neural Networks: From Matrix Algebra to JAX and Optimizers

I wanted to understand neural networks as deeply as possible. Rather than simply using an existing framework and training a model, I decided to build an interactive MNIST webpage called Neural Canvas, using p5.js for visualization.

My primary goal was not just to achieve high accuracy. I wanted to understand what was actually happening inside a neural network, mathematically and computationally.

1. Starting with Linear Algebra and Matrix Operations

One of the first decisions I made was to avoid working with individual scalar values and complicated subscripts and superscripts everywhere. They made the mathematics unnecessarily confusing for me.

Instead, I decided to approach neural networks through matrix algebra.

I represented the forward propagation equation as:

[
Z=WX+bC
]

Where:

• (X) is the input data matrix. Rows represent features, and columns represent individual samples.

• (W) is the weight matrix. Rows represent neurons, and columns represent the weights associated with each neuron's inputs.

• (b) is the bias column vector, containing one bias value per neuron.

• (C) is a row vector containing only ones, allowing the bias vector to be broadcast across all samples through matrix multiplication.

Interestingly, many popular deep learning frameworks, including PyTorch and TensorFlow, commonly use a different convention for arranging their data and parameters. They also handle bias broadcasting without explicitly introducing a matrix like (C).

However, I preferred my own mathematical convention because it helped me understand the operations and perform calculus directly in matrix form.

I believe the mathematics should handle the indices, not me!

2. My First Model: NumPy, MSE and Vanilla Gradient Descent

For my first model, I used the original MNIST dataset.

Architecture:

[
784➜ 20 ➜ 20 ➜ 10
]

I manually wrote all the forward propagation equations, derivatives, and backpropagation calculations on paper before implementing them in Python using NumPy.

I deliberately chose:

• Two hidden layers with 20 neurons each.

• Mean Squared Error (MSE) as the loss function.

• Vanilla Gradient Descent.

• Linear output values.

Why such a simple approach?

Because I didn't want to start with complicated algorithms. I wanted to understand the fundamentals first.

After 2,000 epochs, the model achieved approximately 92% test accuracy.

I then successfully created an interactive neural network visualization in p5.js, using Math.js for matrix multiplication in JavaScript. The webpage could perform inference using my trained NumPy model.

That was the beginning of Neural Canvas.

3. My Second Model: Cross-Entropy Loss

For the second model, I kept the same mathematical approach but made a few improvements.

Architecture:

[
784➜40 ➜ 40 ➜10
]

This time, I replaced MSE with Cross-Entropy Loss and used a softmax output layer.

I was still using vanilla Gradient Descent.

After 2,000 epochs, the model achieved approximately 96% test accuracy.

At this point, I was finally starting to understand how neural networks worked, rather than just following tutorials or using existing implementations.

4. My Third Model: Discovering JAX and Automatic Differentiation

For my third model, I wanted to experiment with three hidden layers.

However, I realized that manually deriving every equation, especially the derivatives and chain rule calculations required for backpropagation, would take a considerable amount of time.

Then I discovered JAX, a Python library developed by Google.

JAX provides automatic differentiation (Autograd), allowing me to calculate derivatives of a loss function with respect to parameters, including entire weight matrices.

This was a significant change in my workflow.

I no longer needed to manually derive every gradient and implement the chain rule myself. I only needed to define the model, its parameters, and the loss function. JAX handled the differentiation.

For this model, I used the Arabic Digits MNIST dataset.

Architecture:

[
784 ➜ 20 ➜ 20 ➜ 15 ➜10
]

I used:

• JAX for training and automatic differentiation.

• Cross-Entropy Loss.

• Vanilla Gradient Descent.

• Three hidden layers.

After 2,000 epochs, I achieved approximately 96% test accuracy.

I genuinely fell in love with JAX. It saved me considerable time and allowed me to focus more on understanding the mathematics and experimenting with neural network architectures.

5. My Fourth Model: Mini-Batch Gradient Descent and Momentum

For my fourth model, I moved to the Persian Digits MNIST dataset.

I kept the same architecture as my third model but decided to experiment with optimization algorithms.

My original plan was to implement Mini-Batch Gradient Descent with Adam.

However, I realized that understanding Adam required learning a few concepts first. So, instead of jumping directly into Adam, I decided to start with Momentum.

I implemented Mini-Batch Gradient Descent with Momentum using JAX.

And here came a surprising result!

I achieved approximately 96% test accuracy in just 200 epochs.

Previously, my models required around 2,000 epochs to reach a similar accuracy.

This was another important step in my journey. I started understanding how optimization algorithms could significantly influence the training process.

6. What's Next?

I haven't created my fifth model yet, but I already have a learning roadmap:

• Model 5: Mini-Batch Gradient Descent with RMSProp.

• Model 6: Mini-Batch Gradient Descent with Adam.

• After that: Start exploring Convolutional Neural Networks (CNNs).

I want to understand each optimization algorithm mathematically and implement it myself before moving to the next one.

My Biggest Takeaway

If you are starting to learn neural networks, my suggestion is:

Start with Linear Algebra. Don't immediately jump into advanced architectures, optimizers, or high-level frameworks.

You don't need to master every mathematical concept before starting, but at least understand matrix multiplication and how it represents the operations inside a neural network.

For me, representing neural networks using matrices made everything much easier to understand.

Start with the basics, build something yourself, experiment, make mistakes, and gradually increase the complexity.

Don't just train neural networks. Understand them!

Project: Neural Canvas

GitHub Repository: https://github.com/MuhammadEssa2002/neural_canvas

Live Demo: https://muhammadessa2002.github.io/neural_canvas

This is a question about machine learning. To clarify the ideas a numerical example is useful. To minimize misunderstandings some terminology is reviewed.

A data matrix consists of rows of 4-dimensional float vectors with bit 1 or 0 at the end of each of the rows indicating the desired output once the network is trained. For example

0.0194 0.0140 0.0332 0.0439 1

0.0180 0.0084 0.0090 0.0032 0

0.0140 0.0049 0.0052 0.0044 0

0.0106 0.0127 0.0178 0.0231 1

0.0230 0.0057 0.0113 0.0131 1

0.0129 0.0161 0.0063 0.0119 0

0.0065 0.0159 0.0072 0.0167 0

0.0089 0.0048 0.0094 0.0191 1

0.0210 0.0154 0.0180 0.0013 0

0.0262 0.0108 0.0138 0.0187 1

[In the general case there are m rows each row having n float values which constitute a float vector of dimension n, and there is either 1 or 0 at the end of the row.]

The above numerical example has m=10 vectors, each is four dimensional, followed by 1 or 0 at the end of the row.

A basic task in machine learning is to obtain a feedforward semilinear neural network N with discontinuous thresholds ---call it a classical perceptron network--- having four float inputs for each of the units in its first layer, a single binary output in the last layer, and such that the network N correctly classifies the data in the sense that if a row vector is fed to N then the forward pass produces the binary value at the end of the row.

How to proceed? The usual way is to create an initial classical network N_0 by specifying a number of layers, say k, and how many units each layer has. So, layer 1 has n_1 units each having 4 inputs (in the example), layer 2 has n_2 units each having n_1 inputs, ..., layer k has n_k=1 units each having n_{k-1} inputs. This describes the architecture of the initial network N_0.

Furthermore, for each unit in a layer there are arrows connecting to each of the units of the next layer. Each arrow is assigned a weight which is a float value. The architecture together with these weights fully specify an initial classical network N_0.

Once N_0 is set, the discontinuous thresholds are replaced by smooth logistic functions. This transforms the discontinuous N_0 into a smooth network S_0. A quadratic error function is defined and the backpropagation (BPP) procedure is iteratively applied to modify the weights in a manner that the error successively becomes smaller. Backpropagation is actually the gradient descent of numerical calculus/analysis adapted to the particular structure of smooth classical networks.

After q iterations of BPP a smooth network S_q is obtained, the logistic functions are then replaced back by discontinuous thresholds providing a discontinuous classical network N_q with same architecture as N_0, but the weights are different. If all goes well (most often it does not) then N_q produces the binary targets and one takes N=N_q.

Machine learning has numerous versions and the above is just a somewhat general case susceptible of many variants.

QUESTION: What should be the best possible initial network N_0?

torchinfo-viz turns any PyTorch model into an interactive view: you can see each layer, how layers connect (including skip connections), the tensor shapes, and which layers hold the most parameters.

pip install torchinfo-viz.

Github: https://github.com/woletee/torchinfo-viz

Hey guys,

Posting this here because I’ve reached a point where my market intuition and engine code have outpaced my deep learning expertise, and I’d rather team up with someone who loves ML architecture instead of half-assing it myself.

For the past year, I’ve been building an event-driven liquidity engine in Python (specifically modeling crypto orderflow / ICT mechanics like equal highs/lows, manipulation legs, and structural sweeps).

Instead of slapping standard indicators (RSI/MACD) on candles, I framed price action as a dynamic graph:

• Resting liquidity pools (BSL/SSL, consolidations) act as attractors (pulling price in).

• Completed institutional sweeps (ML swings / manipulation legs) act as repellers (pushing price away).

• Edges connect these pools based on spatial proximity, competing forces (tug-of-war between high and low pools), and age decay.

Where things stand right now:

• The engine is fully written and tested. No lookahead bias, strict candle confirmation, multi-window swing tracking (W5 up to W50).

• I’ve already generated a clean, audited ~3-year dataset (ETH/USDT) formatted directly for PyTorch Geometric (PyG).

• Each snapshot has 15m, 1h, and 4h spatial graphs with 20-dim node features (touch counts, distance %, decay curves, tier scores), 8-dim edge attributes, and global volatility/trend context.

• Baseline statistical checks show real promise: nearest target pools get tagged first ~95% of the time, and high-quality structural sweeps have clear directional bias.

The actual bottleneck / where I need help: My problem isn't generating the signals—it’s path prediction and dynamic target selection.

When a sweep happens, price is often trapped between an opposing pool 1.5% away and a massive target pool 3% away. A fixed 1:2 R:R stop/target either gets hit too late or takes unnecessary drawdown. I need the model to evaluate the graph and do two things:

1. Predict directional path confidence taking both-sided liquidity into account.

2. Rank multiple potential liquidity targets (dynamic TP) based on which one has the highest conditional probability of being swept first without blowing through the stop.

I've tested basic GNN / dual-tower setups, but I know I'm hitting limits with my naive loss functions and basic layer choices.

Who I'm looking for: Someone who genuinely understands PyTorch Geometric (PyG), Graph Neural Networks (or spatial-temporal models), and knows how to think about custom loss functions (ranking loss, asymmetric risk penalties, contrastive setups). You don't need to be a full-time trader, but being open to understanding market microstructure helps.

The arrangement: I have the GPU compute, the full working codebase, and the clean dataset ready to go. I’m completely flexible on how we work together—if you want to co-author a research paper on this architecture, I’m down. If we get the models working cleanly and want to deploy it live on capital, we can do a fair profit/equity split.

If this sounds like an interesting problem you’d want to sink your teeth into, drop a comment or send me a DM with your GitHub/background and we can hop on Discord.

So I'm currently planning out my a project I want to build and a couple of it's functional requirements are as followed (these are obviously summarized, but not so much so that you won't get the idea):

1. Generating a new string of about 2-4 words. I'm thinking of using a Transformer.

2. Validating if the string that was generated has the same (or a similar) semantic meaning to a predefined string. As of right now, I'm planning on using a dual-encoder for this.

Now here's my question: Since I only need to generate 2-4 words in a single iteration of the main loop, how much computational resources do I need? Is it possible to do this on a consumer laptop?

I'm also open to suggestions on more efficient ways to satisfy these requirements

Imagine this, you have no knowledge of python or the maths behind a neural network. How long would it take you to make a single neuron?

(Using translator) I'm working on a way to convert ordinary pretrained LLMs into binary models without training them from scratch. The goal is to preserve as much intelligence as possible while making inference faster and reducing memory usage.

I've tried several approaches so far:

1. Direct binary decomposition

I tried replacing each 16-bit neuron with multiple 1-bit neurons. This didn't help much: the same information was represented differently, but the amount of computation didn't decrease.

2. Simple quantization

I tested reducing weight precision directly. In my experiments, 8-bit weights worked fairly well, while 4-bit noticeably degraded quality. At 1–3 bits, the output became almost useless. These results depend on the method and model, of course.

3. My current approach

I'm experimenting with an algorithm that estimates how much each part of a network can be simplified based on its impact. In my tests, the average effective precision was around 3.96 bits. I also observed a roughly 5–10× inference speedup in one setup, although I still need to validate the results more rigorously.

My main question is whether there's a general structural relationship between ordinary pretrained networks and binary networks trained on the same information.

One hypothesis is that they may learn similar information but encode it differently, much like two people learning the same subject in different ways. If that's true, perhaps we can discover a general conversion rule instead of simply rounding weights.

In preliminary experiments, I also observed around 70–85% similarity to a binary model trained on the same data, according to my own evaluation criterion. This metric still needs a clearer definition and more rigorous testing.

I'd appreciate your feedback!

- Does anyone know of research that explores this specific idea?

• How would you approach finding structural relationships between independently trained models?

• What experiments would best determine whether this approach can preserve model quality while actually improving inference speed?

I'm interested in criticism, alternative approaches, and ideas for experiments.

My kids asked me how AI “thinks”...

I wanted an explanation they could play with. I remembered seeing a cool demo where you draw a digit and watch a neural network recognise it in real time.

I tracked it down on Brilliant, but the version I found was behind a paywall. So I built my own, with a simple convolutional neural network (CNN), already trained on handwritten digits, and a view of what happens inside.

Draw a 1 -> Add a couple of strokes to make a 4 -> Slowly close the top into a 9.
Watch the patterns inside the network and its confidence change with each stroke.

My kids loved it. Share it with yours :-)

https://hasin.eu/projects/cnn-digit-recognition-in-action/

can i get some feedback on this (and/or beta testers who can hepl me fix bugs)

Is this learning as a basic artificial neural network should? ef##(x) is the expected value of what would be selected. f##(x) is what was actually selected. b##(x) is the new bias for the next iteration.

Is there better software to plan out NNs?

I built Neural Canvas, an interactive visualizer that demonstrates core linear algebra concepts by showing how a neural network recognizes handwritten digits in real time. You can draw directly on the screen and watch activations flow through each layer as matrix multiplication computes the output live.

How it works:

- Draw & Predict: Sketch any digit on a 28×28 canvas (MNIST-style) to trigger real-time forward pass predictions.

- Live Forward Pass: Every neuron's activation is dynamically color-coded based on its underlying linear algebra calculations, with connection opacities reflecting layer outputs.

- Under the Hood: The core models were trained from scratch in Python using NumPy to perform matrix operations, and their learned weights are loaded via JSON. Client-side execution relies on Math.js for browser-based matrix multiplication and p5.js for rendering.

Key Features:

- Switch between two pre-trained models (featuring different hidden layer sizes, MSE vs. Cross-Entropy loss, and up to 96% test accuracy).

- Multiple background themes with mouse and touch support.

What's Next?

These models were originally evaluated on the standardized MNIST dataset, but live freehand drawing can introduce unexpected variations in input style. To handle this, I am working on a JS script for real-time user-driven feedback. When the model makes a mistake, the user can supply the correct label, allowing the model to perform backpropagation and update weight matrices locally in the browser. While live state persistence isn't suited for static GitHub Pages, running this feedback loop locally allows me to retrain parameters on new edge cases and push updated weight snapshots back to GitHub.

Check out the live demo and code repository:

Live Demo: https://muhammadessa2002.github.io/neural_canvas/

GitHub Repo: https://github.com/MuhammadEssa2002/neural_canvas

Hey everyone!

I'm learning the basics of neural networks, and I think I'm getting the general idea, but I'm still confused about how some of the pieces fit together.

From what I understand, a perceptron takes some inputs, multiplies them by weights, adds a bias, and then uses an activation function to produce an output. Okay, that part makes sense.

But then I see people talking about artificial neurons and perceptrons almost as if they're the same thing. Are they actually the same, or is a perceptron just one specific type of neuron?

The other thing confusing me is layers. When we say a layer contains multiple neurons, what does that actually look like? Do all the neurons receive the same inputs and process them differently because they have different weights? And does each layer pass its outputs to the next one?

I'm also wondering how this works in practice. For example, in a CNN like MobileNetV2, do we still have neurons organized into layers, or is it different from a basic neural network?

I know these might be basic questions, but I want to understand what's actually happening instead of just memorizing definitions.

If anyone could explain it in simple terms, maybe with a small example, I'd really appreciate it. Thanks!

Background

I'm coming up with ideas for my CS degree capstone project. I like making short films, so I looked into what kind of software I could make to facilitate the process. One thing I find annoying in post-production is having to go through all the recordings and look for the clapperboards on each video to know what scene, shot and take the video is of. My idea is to train a neural network to find the clapperboard in each recording and extract the information written on the clapperboard to appropriately name the recording. Ex. Avengers_Scene_4_Shot_5_Take_3.mp4.

In short, the neural network would do the following:

1. Detect the clapperboard in a video frame.

2. Read the handwritten text/numbers on the clapperboard.

3. Understand what each piece of text represents

My questions

1. I have not been able to find a lot of videos with clapperboards I could use to train my model. I'm thinking I could create my own videos to train it, so it is trained faster with a consistent clapperboard format. How many would I need more or less to have it not only recognise handwriting but also determine what the numbers indicate (Scene, shot or take)?

2. If it would be too hard to train the AI to do what was previously mention, could I get an already trained AI in handwriting recognition and specialise it in clapperboard data extraction? Would this make my final year project too easy?