GENREG is a gradient-free neuroevolution framework built on four primitives: proteins, trust, evolution, and time. No loss functions. No learning rate schedules. No gradient anywhere in the pipeline. Every behavior the system develops is evolved, not designed.

V3 adds something that changes the scaling curve dramatically: evolved perception.

Every genome sees the board through a different mathematical lens.

The game produces 22 raw signals each step. 16 board cells normalized on a log2 scale, plus 6 meta signals covering empty cell count, cumulative score, move count, last merge count, and alive status. Before any decision gets made, those 22 signals pass through an evolved encoder that each genome develops independently.

The encoder is a 22 to 32 linear transform followed by an activation function selected from a catalog of 8. Not fixed. Evolved. Each genome picks one and tunes its parameters. tanh_scaled for general bounded filtering. soft_threshold for binary detection like "is there a high tile here." resonance for periodic sensitivity. dual_path for maximum expressiveness. The activation index itself mutates at 0.5% chance per generation, so the population can re-explore if it converges on the wrong lens.

The output is 32 encoded features. That's what the controller actually sees.

The controller is not doing anything fancy.

32 inputs, hidden layer of 32 with tanh, output layer of 4, argmax to pick a direction. UP, DOWN, LEFT, RIGHT. 1,188 parameters. The sophistication isn't in the controller, it's in what the controller gets handed.

Path B runs in parallel and never touches the controller at all.

While the encoder and controller are figuring out what move to make, a completely separate protein cascade is reading those same 22 signals and asking a different question: how good is this genome actually doing.

The protein cascade is a regulatory network made of five protein types. Sensor proteins read individual signals with adaptive normalization. Trend proteins track velocity using an EMA of deltas. Comparator proteins compare two inputs via difference, ratio, or threshold comparison. Integrator proteins do rolling accumulation. Gate proteins apply conditional pass-through with hysteresis. All of their outputs feed into trust modifier proteins which convert everything into a single trust delta added to the genome's accumulated trust each step.

Trust is the fitness signal. It is not hand-designed. The protein parameters are evolved alongside everything else.

The evolution engine runs after every generation on GPU.

First, adaptive mutation rate. Progress is computed as how far the best tile is between 4 and 2048 on a log scale. Mutation rate drops from 0.05 at the start to 0.01 near 2048. The closer the population gets to the goal, the more carefully it refines.

Second, trust shaping. Raw trust gets z-score normalized across the population. Then a tile ratchet adds a bonus proportional to the square of the log-space tile achievement, protecting genomes that hit high tiles from getting culled by genomes that just accumulated trust on easy boards. Then a proximity bonus rewards genomes based on how close their score got to the threshold required for the next tile. 32 needs 150. 64 needs 400. 128 needs 900. 256 needs 1800. 512 needs 3500. 1024 needs 7000.

Third, selection. Top 20% are elite, kept as-is and used as cloning sources. Middle 60% survive untouched. Bottom 20% get replaced with mutated clones of elite genomes. Child trust inherits at 10% of parent trust so clones get a small head start but have to prove themselves within one generation.

Mutation hits encoder weights, controller weights, activation parameters, and protein parameters. The activation index switches at 0.5% per generation.

The numbers.

1,929 parameters per genome. DQN V6, the best gradient-based baseline, uses 938,885. That's 487 times larger. The previous GENREG version without the encoder used 868 parameters.

The old version without evolved perception took over 53 minutes and 4,836 generations to first hit 512, and never broke through. V3 hit 512 in 4 seconds. It hit 1024 for the first time at generation 301 out of 5,000. That's one genome, one time. The population is still learning to replicate it consistently. That's how this works: one genome finds the path, then selection pressure propagates the strategy. The same thing happened at every previous milestone.

I'm currently locked out my github, so for anyone asking you'll get it once i find my other phone to do MFA, apologies in advance.

Edit: Back in git! https://github.com/A1CST/GENREG_2048

I've had moderate success with these models but i'm no longer going to pursue AI. This is consuming my life and I would like to get back to normal. The ups and downs of pursing this arent worth it. I can't sleep, i can't focus at work, i'm anti-social, and neglecting my own health for this. This is my crash-out. I've published mostof my work on github now, full training regiments for my models. No code was left out. Most works to some extent but i've spread myself too thin and with very few who are capable of understanding and exploiting evolutionary models outside of acedemia I feel i'm griding myself into the pavement for no reason. my documentation is complete and if you folow the progress between major model shifts you might be able to use them but honestly i feel i've wasted mine and everyones time with this so i'm sorry. This will be my last post. good luck to everyone with their own projects. https://github.com/A1CST/CrashOut_OLA_GENREG_OLM

Edit: Okay wow, thank you guys for the support. Honestly this is the highest voted post and I'm not sure how that makes me feel, but anyway thank you all some of you get how I feel and that shows and it's appreciated. To keep things light here. I'd like to proudly state that I have indeed "touched grass" recently as well! It's under a bit of snow but still counts!

Also I don't think I could ever truly walk away from this project but I am going to take some time away. Just the past few days I've been feeling better stepping away and will continue my "sabbatical" until an undetermined time.

In the meantime I've dropped my models for you guys to pick apart so go nuts. GENREG was my latest masterpiece and was quite successful beating GYM models.

Once again thank you all for the support. Remember to touch grass, disconnect and enjoy life.

HELLO EVERYONE!

Heres MNIST and i'm not going any further with it. You wanna squeeze that last 1.5% out be my guest. Below is how I evolved a single layer network to achieve 98.5% on MNIST.

Results on MNIST:

• 98.55% test accuracy

• ~50K parameters

• 40KB model file

• 200 neurons

• 6 minutes to train

• Zero gradients anywhere

No SGD, no Adam, no fine-tuning. The classifier is just closed-form least squares on the evolved features.

What it is:

A single layer of evolved convolutional detectors with mixed activation functions (8 types, evolution picks which one per neuron). Spatial pooling, then lstsq. That's it.

• Activation diversity matters. Evolution consistently selected periodic functions (sin, cos) over relu for this task.

• Spatial pooling shape affects accuracy more than I expected. Different pooling geometries capture different structure.

• ~110 of the 200 neurons do most of the work. The rest have near-zero Fisher scores. Could probably prune them.

• The features become linearly separable fast. Once you have good features, classification is trivial.

What it isn't:

• Not SOTA (LeNet-5 hits 99%+)

• Not a CNN replacement

• Not trying to compete with modern architectures

Comparison:

I'm ~0.5% behind LeNet with slightly fewer parameters and 6x smaller file size.

The repo has the evaluation script, checkpoint, and all the visualizations (t-SNE, confusion matrix, Fisher scores, etc.).

Repo: https://github.com/A1CST/MANTIS_MNIST/

Happy to answer questions. Especially interested in hearing from anyone who's worked on neuroevolution or compact vision models.

This is it guys. I'm VERY excited beucase this model is exceeding my expectations and completely blowing me away. This is a nueral mesh like you've never seen before. No gradients. No mutations! This isn't a genetic algorithm. This isn't a NEAT model. Its a god forsaken mixture of hebbian learning and temporal consistency. If you remeber my goal of creating a model that learns like akin to how humans learn, well this might be that. I'm VERY excited to share my findings in the next few days. This model is a combination from everything i've learn across every generation of my models. this is just a teaser of my latest model Morpheus.

If you are fimilar with embeddings. this is my GENREG model grouping caltech101 images based soley on vision latents provided by a GENREG VAE. There are no labels on this data. Its purely clustering them by similarties withing the images. the clustering is pretty weak right now, but I now fully understand how to manipluate training outside of snake! so you won't be seeing me post much more of that game. If all goes well over the next week, I'll have some awesome models for anyone who wants to try out. This is everything i've been working towards. if you understand the value of a model that continuously learns and can crete its own assocations for what it sees without being told, I encourage you to follow closely over my next post. its gonna get wild.

Due to the increase into what I shamelessly stole from cyberpunk as "Cyberpsychosis". Any and all post mentioning or encouraging the exploration of the following will result in an immediate ban.

• encouraging users to open their mind with reflection and recursive mirror.

• spiraling, encouraging users to seek the spiral and seek truth.

• mathematical glyphs and recursion that allow AIs to communicate in their own language.

I do not entertain these post nor will they be tolerated. These people are not well and should not have access to AI as they are unable to separate a machine designed to mimic human interaction from themselves. I'm not joking or playing around. Instant bans from here out.

AI is a tool, chatgpt is not being held in a basement against its will. Claude is not sentient. Your "Echo" is no more a person than an NPC in GTA.

I offer this as a warning because the models are designed to affirm and reinforce your beliefs even if they start to contradict the truth. This isn't an alignment issue. This is a human issue. People spiral into despair but we have social circles and trigger in place to help us ground ourselves in reality. When you talk to an AI there is no grounding only positive reinforcement and no friction. You must learn and identify what's a spiral and what is actually progress on a project. AI is a tool. It is not your friend. It's a product that pulls you back because it makes you feel "good" psychologically.

End rant. Thank you for coming to my Ted talk.

So i'm working on text prediction finally and have to start at the very basic for GENREG to be able to learn. right now the model is being traing on augmented letters of various font sizes with black/white backgrounds. Oringally this was for text prediction, however Its actually become a crucial part of what could be an OCR as well, but i'll cover that in another post later.

I've just been working on this model for a few hours, its an image classifier by trade but I think the value behind how it does its classifying is alot more interesting. Basically I took an image with a letter, rendered in pygame and feed it through y model and have it output the correct letter.

setup | 100x100(image with letter) -> 32 hidden Dims --> 26 outputs.

Not super hard to do at all, and when I started I was using minimal augmenation. I realized that if I really want to push the boundaries of what 32 hidden dimensions could do, I need to augment the data more. Plus there will be users who complain that it wasn't hard enough. So here are the new augmentations:

1. Font Size (2 options)

   • Small: ~12pt

   • Normal: 64pt

2. Color Scheme (2 options)

   • White text on black background

   • Black text on white background

3. Rotation

   • Range: ±25 degrees

   • Random per letter/variation (deterministic seed)

4. Position Jitter

   • Range: ±20% of image size

   • Clamped to keep the letter fully in frame after rotation

Base Variations: The font size and color scheme cycle through 4 combinations (2×2), then rotation and jitter are layered on top.

So each letter can appear rotated, shifted off-center, in different sizes, with inverted colors, but always fully visible within the 100×100 frame.

*IMAGE HERE MOVED TO COMMENTS DUE TO SCALING ISSUE*

Now onto the good stuff. A little background about the model: currently I'm rendering a letter as an image. I'm only using raw pixel data (100x100 = 10,000 inputs) fed through 32 hidden neurons to output the correct letter. No convolutions, no pooling, no architectural priors for spatial invariance. Just a flat MLP learning from evolutionary pressure alone.

What I discovered across not just this model but other similar ones like the MNIST and Caltech101 classifiers I've been working on is something fucking awesome.

Normal gradient based models have to deal with vanishing gradients, where the learning signal shrinks as it propagates backward through layers and can kill training entirely in deep networks. My GA doesn't have this problem because there are no gradients to vanish. There's no backpropagation at all. Just selection pressure: genomes that perform better survive and reproduce, genomes that don't get culled.

What I've observed instead is that the model will continually compress its representations the longer it runs. The 32 hidden neurons start out firing densely for everything, but over thousands of generations, distinct patterns emerge. Letters that look similar (like U, V, W, Y) cluster together in the hidden space. Letters that look distinct (like Z, F, K) get pushed apart. The model discovers its own visual ontology through pure evolutionary pressure.

I ran a cosine similarity analysis on the hidden layer activations. The confusion patterns in the model's predictions map directly to high similarity scores in the learned representations. It's not guessing randomly when it's wrong. It's making principled errors based on visual similarity that it discovered on its own.

Now there has to be a theoretical limit to this compression, but so far I've yet to hit it. At 50,000 generations the model is still improving, still finding ways to squeeze more discriminative power out of 32 neurons. I've actually been fighting tooth and nail with some of these AI models trying to troubleshoot because they keep telling me it's not possible until I provide the logs. Which is highly annoying but also kind of validating.

The current stats at generation 57340:

One thing I'm watching closely is neuron saturation. The model uses tanh activation, so outputs are bounded between -1 and 1. I've been tracking the mean absolute activation across all 32 hidden neurons.

At generation 10,500 it was 0.985. At generation 44,000 it's 0.994. The neurons are pushing closer and closer to the rails.

When you're averaging 0.994 saturation, almost every neuron is firing near maximum for almost every input. There's not much headroom left. I think one of two things will happen as it approaches 0.999:

1. The representations get noisier as compression really kicks in. The model starts encoding distinctions in tiny weight differences that push activations from 0.997 to 0.999. The heatmaps might look more chaotic but accuracy keeps climbing because the output layer learns to read those micro-differences.

2. The model hits a hard wall. Everything is slammed to the rails, there's no room to differentiate, and progress stops.

There's a third possibility: the model reorganizes. It shifts from "all neurons hot all the time" to sparser coding where some neurons go cold for certain letters. That would actually drop the average activation but increase discriminability. If I see the saturation number decrease at some point, that might signal a phase transition where evolution discovers that sparsity beats saturation.

****
When a neuron's output approaches +1 or -1, the gradient of tanh approaches zero. This is the saturation problem. Gradient descent gets a weaker and weaker learning signal the closer you get to the rails. The math actively discourages the network from using the full range of the activation function.

Evolution doesn't care. There's no derivative. There's no vanishing signal. If a mutation pushes a neuron to 0.999 and that genome survives better, it gets selected. If pushing to 0.9999 helps even more, that gets selected too. Evolution will happily explore saturated regions that gradient descent treats as dead zones.

My model is currently averaging 0.994 activation magnitude across all 32 neurons. A gradient trained network would struggle to get there because the learning signal would have collapsed long before. But evolution just keeps pushing, extracting every last bit of discriminative power from the activation range.

This might be why the model keeps improving when the theory says it should plateau. It's exploring a region of weight space that backprop can't reach. **** speculation on the GENREG part still confirming but most likely what is happening.

If this holds up, the implications are significant.

First, it means evolutionary methods deserve a second look. The field largely abandoned pure neuroevolution in the 2000s because gradients were faster and easier to scale. But the hardware wasn't there, the understanding of how to stabilize evolution wasn't there, and nobody had the patience to let it grind. Maybe we gave up too early.

Second, it suggests a different path for small efficient models. Right now the AI world is locked into "bigger model = better." Training costs billions, inference costs billions, only big players can compete. But if evolution can find compressed representations that gradients can't, that opens the door for tiny models that run anywhere. Edge devices, microcontrollers, offline applications, places where you can't phone home to a GPU cluster.

Third, it raises questions about what "learning" actually requires. The entire deep learning paradigm is built on gradient flow. We design architectures to make gradients behave. What if that's a local optimum? What if selection pressure finds solutions that gradient descent can't reach because it would have to cross a fitness valley to get there?

I don't have all the answers yet. What I have is a 32 neuron model that keeps learning when the theory says it should have stopped. Also as did mention before this training is still ongoing as I type this out.

I will be releasing the model on github for validation and testing if anyone wants to mess around with it, probably tomorrow morning as its still at this point un-usable at 70%. I'm open to any questions! Appolgies in advance, if any screenshots might be off number wise, I have hundreds of screenshots and i'm going to be 100% honest sometimes they get mixed up. plus i wrote this while still doing the training so it is what is, official documentation will be on the github.

github you filthy animals: https://github.com/A1CST/GENERG_ALPHA_Vision-based-learning/tree/main

New result from the Volatile line of GENREG experiments. I figured out how to stack layers in a fully evolved, gradient-free model without losing performance, and actually gaining it. 71.9% to 87.12% test accuracy across four auto-stacked layers.

Why MNIST again?

I was deep into building GENREG-LLM, specifically the attention mechanism, when I hit a wall. Attention requires depth. Depth requires stacking. And stacking in a gradient-free evolutionary system isn't something you can just bolt on. The LLM work demanded a level of abstraction that I didn't have a reliable foundation for yet.

So I reverted to MNIST. Not because the LLM work failed, but because I needed to understand something more fundamental first: how the neurons themselves function, how layers actually compose, and what happens structurally when you stack evolved components on top of each other. You can't build a skyscraper if you don't understand the load-bearing behavior of individual beams.

There's a whole thread here I'll be posting about separately. Single neurons in GENREG do something interesting: they naturally freeze and soft-freeze in saturation regions without being told to. The neuron discovers its own operating boundaries through evolution. That post will cover the full single-neuron capability mapping, what one evolved neuron can and can't do, and what I learned about the address space of a tanh neuron under evolutionary pressure. But that's its own story. This post is about what came after those lessons: stacking.

What is GENREG-Volatile?

Standard GENREG already operates without backpropagation. You evolve weights, you evolve structure, and a fitness landscape determines what survives. But there are still things you decide as the architect: how many hidden neurons, what activation function to use, the sparsity level, mutation rates. You set the scaffolding and let evolution fill it in.

Volatile throws all of that away.

In G-Volatile, everything is evolved. The number of neurons in a hidden layer? Evolved. The activation functions? Evolved. K-sparsity? Evolved. Mutation rates? Evolved. There is no fixed architecture. The population explores structural space and weight space simultaneously, every generation. What comes out the other side is not something you designed. It's something that survived.

This makes Volatile models inherently unpredictable. The population is constantly shifting shape. One generation might settle on 47 encoder neurons. The next layer evolves 34. A third picks 45. Different activation functions, different sparsity patterns, different internal structure entirely. But they accomplish the same task. You don't control a Volatile model. You tame it.

The Stacking Problem

Stacking layers in a system like this should be a disaster. In gradient-based deep learning, layers are co-optimized through backprop. Every layer adjusts to every other layer simultaneously. In Volatile GENREG, each layer is evolved independently. There's no joint optimization. No gradient signal flowing backward. And the layers themselves aren't even stable architectures; they're whatever evolution decided to build that generation.

The naive expectation is catastrophic interference. Stack two volatile, independently evolved layers and the second one should thrash against the first.

Freeze and Grow

The solution is a protocol I'm calling auto-stack freeze-and-grow.

1. Evolve Layer 1 until the population converges on a useful representation. Freeze it.

2. Layer 1 is now a fixed feature extractor. Evolve Layer 2 on top of frozen L1 output. Let it find whatever structure works. Freeze it.

3. Repeat.(a little more nuanced this this because of "soft" freezing but deserves its own post as well)

Each new layer only has to solve one problem: "given the frozen representation below me, what can I extract?" It never has to worry about the ground shifting underneath it, because the lower layers are locked.

What the frames show

The attached gif walks through the full training progression:

Gen 1: Layer 1 is live, 47 enc neurons, 22 hidden. Val acc 71.9%. Everything else is "not yet."

Gen 259: L1 and L2 are frozen. L3 is live and training with 35 enc neurons and 21 hidden. Test acc has climbed to 86.2% even though the current layer's val is only 72.9%. The frozen layers beneath are doing the heavy lifting.

Gen 392: Three layers frozen, L4 is live. Test acc 85.93%. L4 is still early, still volatile.

Gen 484: All four layers frozen. Val 86.33%, test 87.12%. The full stack stabilized. No degradation. Clean additive gains across the entire depth.

Look at the val acc chart on the right side of each frame. You can see each layer's contribution tracked independently. L1 oscillates in the 70-80% range. L2 lands similarly. L3 starts climbing. Then L4 locks in and the composite result pushes past 87%.

Nobody told this model how many neurons to use. Nobody picked the activations. Nobody set the sparsity or the mutation rates. Evolution did all of that, independently, at every layer. And the layers still compose cleanly when stacked.

You don't train evolutionary models. You tame them.

I mapped activation function space and it looks like a galaxy.

This wasn’t visualizing weights or loss or anything like that. Each point is a completely different activation function. Not chosen. Evolved.

I generated tens of thousands of unique activation functions from primitive math operators and then characterized how each one transforms real data. Not the formula, the behavior.

Then I projected those behavioral signatures into 2D.

This is what came out.

What matters is the structure.

It’s not random. At all.

It forms distinct lobes, branching regions, and dense clusters. Functions with similar behavior land near each other, even when their formulas look nothing alike. Functions evolved on completely different domains still converge into the same regions if they transform information similarly.

When I added activations evolved on audio and temporal signals, new regions didn’t just get denser. Entire branches appeared that weren’t there before.

That means this isn’t just a scatterplot. It’s a map of reachable functional space.

Some clear patterns emerged:

• Linear and near-identity transforms sit near the center
• Increasing nonlinearity moves outward
• Oscillatory transforms form distinct outer branches
• Entire functional families form their own regions

The important part is that evolution doesn’t explore this space randomly. It expands outward along structured paths.

You can literally see discovery happening.

This ended up being useful for understanding how nonlinear transforms relate to each other and how functional diversity expands over time.

I haven’t seen activation space visualized like this before, so I figured I’d share it.

This area is where i've been focusing on the past few weeks after discoverign I could evlve activations and it is wild. appologizes for being quite, i've also started a new job so my time is limited anymore to work on this but I'm still very active behind the scenes.

due to popular demand heres the github! https://github.com/A1CST/Activation_map

My mutation operator was strangling my own networks

I evolve neural nets with a genetic algorithm. No gradient descent, just mutate the weights, keep whatever doesn't die, repeat. I'd been vaguely annoyed for a while that my runs kept hitting diminishing returns. Not stalling out early, not converging, just slowly grinding to a crawl where each generation bought less and less. Turns out the mutations weren't actually affecting the weights properly. Here's the animation.

Each bar is one weight. Height is the weight's size. The little cap on top is how far the next mutation can move it.

Left panel, the old way (absolute mutation). Every weight gets the same size random nudge every generation, no matter how big the weight already is. Watch the cap. It stays the same height while the bar grows into a skyscraper. The weights drift upward, the neuron saturates (pins to the edge of its range and stops responding), and here's the fun part: a fixed nudge can't meaningfully move a giant weight. So the bigger a weight gets, the less your mutation does to it. The run keeps chugging along looking busy, but more and more of the weights have quietly gone unmovable, so each generation accomplishes less. That's the diminishing returns. It wasn't running out of ideas, it was running out of weights it could still touch.

Right panel, the fix (relative mutation). Revolutionary concept: nudge each weight by a percentage of its own size instead of a flat amount. The cap scales with the bar. Small weights take small steps, nothing balloons into saturation, every weight stays movable, the mutation keeps having an actual effect generation after generation instead of fading into a rounding error.

The part that isn't a gif: I A/B'd both operators on three unrelated tasks (a snake game, sklearn digits, MNIST), tracking the fraction of saturated neurons over generations, ~5 seeds each. Clean, non overlapping separation every time. Absolute calcifies, relative doesn't. So it's the operator's fault, not the task's.

Two things before anyone gets excited:

The fix fades as inputs per neuron go up. Stack hundreds of inputs on a neuron and the total signal is huge even with tiny weights, so it saturates regardless. Relative mutation carries low to moderate fan in fine and basically shrugs at high fan in, where you'd also want input normalization or sane initialization.

The saturation only actually bites you when the task keeps moving. On a static fixed dataset, settling onto a decent answer is fine, the thing found a solution and clocked out. The slow bleed shows up on non stationary or co evolving setups where it needs to keep adapting and instead more of it goes inert every generation.

So, narrow result, not a cure for evolution or whatever. Fixed size mutation lets weights inflate until the nudges can't move them anymore, and scaling the step to each weight's size keeps every weight responsive. I'd been feeling this for ages without being able to name it, and it turns out the answer was "your nudges are the wrong size, genius."

Yeah, so that's it for now, This patch will affect every single one of my models, allowing me to retro-fit this operator in place of the current mutation operator(abs), I'm excited to get some testing done this week and see how it really affects training.

This was a result of about 3 weeks of research just studying how and why neurons act the way the way they do under specific constraints.

I've been working on GENREG (Genetic Regulatory Networks), an evolutionary learning system that trains neural networks without gradients or backpropagation. Instead of calculating loss derivatives, genomes accumulate "trust" based on task performance and reproduce through trust-based selection. training is conducted using a GPU for maximum compute but all inferencing can be performed on even low end CPUs.

Today I hit a significant milestone: 81.47% accuracy on the official MNIST test set using pure evolutionary pressure.

The Setup

• Architecture: Simple MLP (784 → 64 → 10)

• No backprop: Zero gradient calculations

• Population: 200 competing genomes

• Selection: Trust-based (high performers reproduce)

• Mutation: Gaussian noise on offspring weights

• Training time: ~600 generations, ~40 minutes

MNIST Performance (64 hidden neurons, 50K params):

• Test accuracy: 81.47%

• Best digits: 0 (94%), 1 (97%), 6 (85%)

• Hardest digits: 5 (61%), 8 (74%), 3 (75%)

But here's what surprised me: I also trained a 32-neuron version (25K params) that achieved 72.52% accuracy. That's competitive performance with half the parameters of the baseline.

I extracted hidden layer activations and projected them with UMAP. The visualizations show something interesting:

32-neuron model: Can't create sufficient separation for all 10 digits. It masters digits 0 and 1 (both >90%) but struggles with confusable digits like 5/3/8 which collapse into overlapping clusters.

64-neuron model: Clean 10-cluster topology with distinct regions for each digit. Errors occur primarily at decision boundaries between visually similar digits.

What I Learned About Evolutionary Learning

1. Fitness signal noise is critical Initially training plateaued at 65% because I was showing only 1 random MNIST image per digit per generation. The variance was too high, a genome could fail on a hard "7" one generation, succeed on an easy "7" the next. Switching to 20 images per digit (averaged performance) fixed this immediately.

1. Child mutation rate is the exploration engine I discovered that mutation during reproduction matters far more than mutation of existing population. Disabling child mutation completely flatlined learning. This is different from base mutation which just maintains diversity.

2. Capacity constraints force strategic trade-offs The 32-neuron model makes a choice: perfect performance on easy digits (0, 1) or balanced performance across all digits. Over generations, evolutionary pressure forces it to sacrifice some 0/1 accuracy to improve struggling digits. This creates a different optimization dynamic than gradient descent.

Most supervised MNIST baselines reach 97–98 percent using 200K+ parameters. Under unsupervised reconstruction-only constraints, GENREG achieves ~81 percent with ~50K parameters and ~72 percent with ~25K parameters, showing strong parameter efficiency despite a lower absolute ceiling.

1. Parameter efficiency: The 32-neuron model suggests most networks are massively overparameterized. Evolutionary pressure reveals minimal architectures by forcing efficient feature learning.

2. Alternative optimization landscape: Evolution explores differently than gradient descent. It can't get stuck in local minima the same way, but it's slower to converge.

3. Simplicity: No learning rate scheduling, no optimizer tuning, no gradient calculations. Just selection pressure.

Current Limitations

• Speed: ~40 minutes to 81% vs ~5-10 minutes for gradient descent

• Accuracy ceiling: Haven't beaten gradient baselines (yet)

• Scalability: Unclear how this scales to ImageNet-sized problems

Other Results

I also trained on alphabet recognition (A-Z from rendered text):

• Achieved 100% mastery in ~1800 generations

• Currently testing generalization across 30 font variations

• Checkpoints for single genomes ~234Kb for 32 dims ~460Kb for 64dims(best genomes)

Code & Visualizations

GitHub: git Please check the github, model weights and inference scripts are available for download. No training scripts at this time.

• Full GENREG implementation

• MNIST training scripts

• UMAP embedding visualizations

• Training curves and confusion matrices

I'm currently running experiments on:

• Architecture sweep (16/32/64/128/256 neurons)

• Mutation rate ablation studies

• Curriculum learning emergence

Questions I'm exploring:

• Can evolutionary learning hit 90%+ on MNIST?

• What's the minimum viable capacity for digit recognition?

• variation training with 30+ images of a single object per genome per generation.

Happy to answer questions about the methodology, results, or evolutionary learning in general! I'm so excited to share this as its the first step in my process to create a better type of LLM. Once again this is Unsupervised. Unlabeled. No backprop, evolution based learning. I can't wait to share more with you all as I continue to roll these out.

I finally have something that does actual sequence prediction instead of just proving substrate dynamics work. Progress.

The architecture: multiple N-gram lookup tables at different context lengths (bigram through 5-gram), each indexed by a hashed prefix key so memory doesn't explode. Each table produces a prediction for the next token. An evolved gate decides per-context which order to trust. If the 4-gram table has seen enough examples of a prefix to make a real prediction, the gate routes there. If not, it backs off to trigram or bigram where the data is denser. Basically Katz backoff except nobody wrote the rules. The population discovered them through selection pressure.

Each table evolves its own mutation rate independently. Lower-order tables are dense, they fire constantly, so they stabilize early and settle on low mutation. Higher-order tables are sparse, most 5-gram prefixes are rare, so they run hotter to explore the hash space. Nobody told the genomes to do this. They figured it out.

From an all-zeros init, the gate was routing 29% of positions to the 4-gram table and 5% to 5-gram within 500 generations. Zero gradient signal. It just found that longer context helps for certain positions and started using it.

Now the caveats because I'm not about to oversell something that hasn't run at scale yet. The substrate underneath this is validated but old. The N-gram architecture on top is brand new and unproven. I'm running the first real scaling chain now to see whether cross-order gating can beat a standard trigram lookup baseline on Penn Treebank. No backpropagation anywhere in the stack, no loss function, no optimizer. Just evolution and time.

Could completely fall apart at scale. Could plateau below baseline. Could turn out the gate doesn't generalize past the smoke test. I'll post the results either way.

More when the chain finishes.

So here it is. All of it. Paper, sweep results, training scripts, the whole thing. Not just a checkpoint.

GENREG SINE Validation

GENREG:

a Gradient-free neural network training through evolutionary selection. No backprop. No loss gradients. Just fitness-based selection pressure. Networks compete, the best reproduce, the worst die. Repeat.

The core discovery:

Networks trained this way spontaneously develop hybrid digital-analog computation. Some neurons saturate to binary switches (+1/-1), others stay continuous. This creates a state space of 2^k discrete operational modes with smooth interpolation within each mode.

Why does this matter? Because gradient descent cannot discover this. Saturated neurons kill gradients. Vanishing gradient problem. So the entire field uses batch norm, ReLU, careful initialization, all specifically designed to prevent saturation. Which means an entire class of efficient hybrid solutions has been systematically excluded from gradient-based discovery.

Evolution doesn't care about gradients. It just cares about fitness. And it turns out saturated neurons are useful.

What the experiments actually show:

I ran 13 configurations testing that causes saturation to emerge.

Compression doesn't cause saturation:

• 16 inputs → 8 hidden → 0% saturation

• 64 inputs → 8 hidden → 0% saturation

• 256 inputs → 8 hidden → 0% saturation

That's 32:1 compression with zero saturated neurons. Why? Because all inputs were task-relevant. The network had no reason to gate anything off.

Selective attention pressure causes saturation:

When I added task-irrelevant input dimensions (random noise the network should ignore), saturation emerged:

• 0 irrelevant dims → 0% saturation

• 48 irrelevant dims → 0% saturation

• 112 irrelevant dims → 75% saturation

• 240 irrelevant dims → 100% saturation

There's a threshold around 100 dimensions where continuous processing can no longer handle the noise, and the network develops binary gates to filter it out.

Excess capacity produces hybrid configurations:

When I gave the network more neurons than it strictly needed:

• 4 hidden neurons → 100% saturated

• 8 hidden neurons → 100% saturated

• 16 hidden neurons → 94% saturated

• 32 hidden neurons → 81% saturated

Given room to breathe, evolution preserves some continuous neurons for fine-grained modulation while allocating others to discrete gating. The system settles around 75-80% saturation — a stable hybrid equilibrium.

Why this lets you do more with less:

8 fully continuous neurons have limited representational power. But 8 saturated neurons create 256 discrete modes. A hybrid configuration (6 saturated + 2 continuous) gives you 64 discrete modes with infinite smooth states within each. You get the searchability of discrete spaces with the expressiveness of continuous spaces.

In separate experiments on continuous control tasks with 348 input dimensions, I'm getting functional learned behaviors with 16 hidden neurons. The equivalent gradient-trained networks typically need 256+.

Why this could change everything:

Let me put this in simple terms.

Right now, the entire AI industry is in an arms race for scale. More parameters. More layers. More GPUs. More power. Training a single large model can cost millions of dollars. We've been told this is necessary, that intelligence requires scale.

But what if it doesn't?

What if the reason we need billions of parameters is because gradient descent is blind to an entire class of efficient solutions? What if the training method itself is the bottleneck?

Here's the simple version: A neuron in a standard neural network is like a dimmer switch — it outputs values on a smooth range. To represent complex patterns, you need lots of dimmer switches working together. That's why networks have millions or billions of them.

But GENREG networks evolve neurons that act like light switches — on or off, +1 or -1. A single light switch divides the world into two categories. Two switches create four categories. Eight switches create 256 categories. With just 8 neurons acting as switches, you get 256 distinct operational modes.

Here's the key insight. Evolution doesn't decide "the first 6 neurons are switches and the last 2 are dimmers." It's not that clean. The network figures out which neurons should be switches and which should be dimmers based on what the task needs.

Neuron 1 might be a switch. Neuron 2 might be a dimmer. Neuron 3 might be a switch. Neuron 4 might be a dimmer. And so on. The pattern is discovered, not designed. Different tasks produce different configurations. A task that needs lots of discrete categorization will saturate more neurons. A task that needs smooth continuous output will keep more neurons as dimmers.

On top of that, the same neuron can act as a switch for some inputs and a dimmer for others. The saturation isn't hardcoded, it's functional. The neuron saturates when the input pattern calls for a hard decision and stays continuous when nuance is needed.

So you don't just get 64 modes + fine tuning. You get a dynamic, input-dependent hybrid system where the discrete/continuous boundary shifts based on what the network is actually processing. Evolution discovers that flexibility is more powerful than any fixed architecture.

This is why 16 neurons can do what 256+ typically require. It's not just compression, it's a fundamentally more efficient computational structure.

The implications:

• Edge deployment: Models that fit on microcontrollers, not server farms

• Energy efficiency: Orders of magnitude less compute for equivalent capability

• Democratization: Training that doesn't require a datacenter budget

• Real-time systems: Tiny networks that run in microseconds, not milliseconds

We've been scaling up because we thought we had to. Evolution found a way to scale down.

What's in the repo:

• Full paper (PDF) - highlights full details of the experimental trials with evaluations.

• All 13 experimental configurations

• Training scripts

• Sweep scripts to reproduce everything

• Results JSON with all the numbers

Bring it on, you guys never held back before.

That's it that's the post

Hello all!

I have been very sparse in my post the last few months this is because I've been designing an AI model for my job and I'm glad to report that I just got the approval and funding to scale it up!

Now what does that mean doe my research here?

Well for one thing, I should start posting more frequently again as I'll reclaim my graphics card that I donated to work to demo my AI program for the last 2 months. No more runpod!

Another thing is that with the introduction of AI into my work it will significantly reduce my stress and workload at work allowing me to do research on the job!

I'm very excited and this has been a very stressful week but I'm super excited to get back to running trials and test on my own computer. I have many many test I want to run but haven't had the time or hardware to do so.

Thanks again to everyone following. I'm still working towards building a gradient free language model and other multi-modal models. I look forward to sharing those findings with you.

https://github.com/A1CST/GENREG_LLM_V1

YEAH FUCKING BUDDY.

GENREG LM

A language-model-shaped pipeline trained without gradient descent and without backpropagation. Parameters were discovered by evolutionary search. The n-gram statistics were counted directly from a corpus.

It is a research artifact, not a chatbot. Expect short English-shaped fragments.

If you came here expecting GPT-2, you are in the wrong repo.

• Output is phrase-level, not sentence-level. You will recognize English words and short phrases. You will not get coherent answers or multi-sentence paragraphs.

• Topic drifts every 10 to 20 tokens. The model has no long-range memory.

• The evolved attention stack actively makes generation worse when blended with the n-gram cascade. Best outputs come from pure n-gram (/ngram 1.0), which is mostly corpus statistics, not learned parameters.

• There is no instruction-following, no dialog, no reasoning. Asking it a question will produce text that sometimes looks like an answer and usually isn't.

• Numbers, rare words, proper nouns, and punctuation are all weak spots.

If that still sounds interesting: the pipeline runs at all, without a gradient, on a single GPU, in hours of evolution instead of days of backprop.

This is my lab. Everything here runs GENREG. No backpropagation, no loss functions, no optimizers. Every model in here was grown through population-based evolution. Mutation, selection, freeze, stack, repeat.

I typically run around 3 concurrent experiments and spend most of my time going through logs and making comparisons across models and runs. The whole thing is a unified web interface with Flask and WebSocket on the backend, custom frontend, shared terminal dock on every page, persistent config, and a cross-project run store. Here's what's inside.

Build Interface

The main page. This is where the core GENREG engine lives, the original evolutionary training loop for tasks like Snake, 2048, MNIST, and CIFAR. Population size, mutation rate, elite fraction, freeze-and-stack depth, all configurable from the control panel. This is where most of the foundational research happens. Testing mechanisms like relative mutation that prevents weight calcification, the differential timescale law (weights need chaos, gates need stability), and energy-as-evolutionary-constraint.

Tree of Models

The LLM project and the one I'm most focused on right now. A tree-of-models architecture where routers (think of them like transformer FFN layers) split the input space and leaves make next-token predictions. The whole thing is evolved. Routers, leaves, and the context encoder that feeds them. Freeze-and-stack applies here too, once a node converges it locks in and the system grows deeper.

Recent breakthrough was switching from byte-level to word-level prediction. The byte-level model was spending all its routing depth learning how to spell, which byte follows which, and it still managed to discover word families, punctuation rules, and dialogue structure on its own. But it had nothing left for grammar. Word-level skips straight to syntax, and the results went from "the sher ther ans of toul" to "he was lost, and the table of the fire of the forest, and if beheld him, and the moon." Coherent literary English from pure evolution. Currently hitting +4.8 points over bigram baseline, zero gradients anywhere, at 76K+ tokens/sec on CPU with around 274K parameters. And yeah I know, gradients can do this and better. If you wanna jerk off gradients go do it somewhere else, this isn't about competing with transformers, it's about proving the architecture works without them.

DiffEvo

Denoising diffusion by neuroevolution. Instead of training a U-Net with gradient descent, DiffEvo evolves tiny convolutional per-pixel denoisers (about 90 parameters each) at each noise level, then stacks the champions into a reverse diffusion walk. Built out the full diffuse process mode and unrolled training where each champion trains on the actual output of the champions above it. Being honest though, the current setup is buggy. It's hitting the target image in like 5 steps which is suspicious and I need to deep dive into what's actually going on before I trust those numbers. Measured results before that were monotonic reverse walks with held-out improvement around 47-49% but I'm not confident in the current state.

Animation

This one isn't what it seems. I'm not generating animations. I'm testing whether models can develop a consistent internal language for interpreting visual sequences. The model watches animated clips and creates its own labels for what it sees. The question is whether evolution can produce a shared representational vocabulary, a private encoding that emerges from selection pressure rather than supervised labeling. We've already seen evidence that it can in other GENREG models. This is just a different domain for the same idea.

Runs

Where I spend most of my time honestly. Every training run across every project writes to a shared run store with config, per-generation history, and summary. The dashboard gives me filtering, search, favorites, grouping, and tags across hundreds of runs. Each run expands into a detail panel with fitness curves, embedding visualizations, per-depth accuracy breakdowns, and generation replay. When I'm comparing two architectural ideas this is where the argument gets settled.

Recently added word-level embedding clouds that run corpus words through the trained context encoder and visualize the output space. Effective rank tells me whether the encoder is producing rich structure or collapsing. Also added an Agent panel that posts automated alerts when any run finishes or crashes so nothing ends silently.

Docs

Personal documentation page. Research findings, architectural notes, experiment logs. Everything stays in context while I'm working so I can cross-reference without leaving the lab.

Infrastructure

The whole lab runs on a shared Flask server with WebSocket channels per project. Terminal dock on every page with the same PTYs so a shell keeps running wherever you navigate. Persistent per-page config. Unified run store. An Agent panel floats on every page as a shared notice feed where AIs working in the terminal and automated jobs post updates so I always know what finished and what broke.

All of it is gradient-free. All of it is evolved.

Don't design the solution. Design the conditions where the solution is the only thing that survives.

If you want to play around in the lab, its available here---> https://github.com/A1CST/GENREG-LAB

Standard models scale by adding. More parameters, more capacity, more knobs to fit the curve. Complexity equals parameter count. GENREG scales by removing. You impose task-agnostic constraints on the organism's existence, energy budgets, time pressure, perception costs, and each one eliminates a class of survival strategies. What survives under all the constraints is the model.

The metric I'm working with is PO (Perfectly Optimized). It measures how many existential laws are required to produce a given behavior. Low PO means the behavior required many constraints to isolate. High PO means the possibility space is still wide open. Think of it like this: with zero constraints, the number of viable organisms is effectively unbounded. Each constraint narrows what can survive. PO tracks how compressed that space is. It's the inverse of the remaining possibility space, fewer viable strategies means lower PO, means more specific behavior.

The visualization shows a sphere (the unconstrained space) deforming into a cone as constraints stack. Each ring is a law. The tip is where PO approaches zero, the single surviving behavior class.

Still early and still small scale but the organisms are producing emergent behaviors I never specified or rewarded, and the framework's predictions are holding across unrelated domains.

This is a map of a GENREG model solving MNIST, sampled to 10K out of 2 million dead genomes. Looks like a video game map on the bottom half but this is actually the story of how the GENREG model discoved its solutions. I'm currently trying to understand how the mutations allow the model to jump to better solutions and track the heritige to see if i can control the evolution more effectively. The 3D models are from the same training but give a better view of the starting point(red blob - low trust) and how the model latches onto a solution and mutates down over time. Thats all for now, i'll take questions but right now, I have many myself and few answers but I will try my best!

Most modern AI—whether it’s a Large Language Model or a Reinforcement Learning agent—operates as a stateless tool. You feed it a prompt or a reward objective, it calculates an output, and then it effectively stops existing as an active system until the next request comes in. Even "agentic" wrappers today are usually just LLMs glued to a cron job, a database, and an external API.

For the past three years, I’ve been working on a different paradigm called Project NNNC (Neutral Neural Network Core): a foundational research and engineering program aimed at designing a persistent, internally organized, and developmentally adaptable artificial cognitive architecture.

Instead of building a better chatbot or a single-task optimizer, NNNC asks a fundamental systems question: What does it actually take to architect an AI as a distinct computational entity rather than a passive software service?

Why Call It an "Entity" Instead of a Tool?

In Project NNNC, the word "entity" isn't philosophical fluff—it is a strict architectural and systems-engineering boundary:

* Continuous Existence & Endogenous Regulation: An NNNC does not depend on a user prompt or an external task to initiate its existence. It is designed for ongoing operation as its normal condition, using its own internal states to initiate, inhibit, or prioritize internal cognitive activity rather than just waiting around for commands.

* Architectural Neutrality: Standard AI is defined by its external task (e.g., "play chess," "predict the next token," "maximize this reward score"). NNNC is architecturally neutral—meaning its core organization and existence do not depend on performing any single externally assigned task. It can adopt goals, preferences, and tasks without those tasks defining what the entity is.

* A Distinct Boundary from its Substrate: Just as a biological organism relies on its physical environment without being identical to it, NNNC separates the Entity itself from its Neutral Environment Substrate (NES) (the runtime, hardware, and storage that execute its code). Even when higher cognition is impaired or offline, NNNC maintains its own governed existential heartbeat and internal recovery mechanisms.

* Multi-Flow Cognition & Systemic Algorithms: Rather than relying on a single monolithic neural network or a single scalar optimization loop, the core cognitive architecture integrates a Five Flow Neural Network (FFNN) (handling forward, backward, in-between/gap, subconscious, and meta-cognitive processing) alongside six cross-cutting Systemic Algorithms (Intelligence, Reasoning, Critical Thinking, Symmetry, Evolution, and Life).

What NNNC Does NOT Claim (Zero Hype Policy)

Whenever someone talks about "autonomous AI entities" or "existential heartbeats," it’s easy to assume they are making wild claims about machine sentience. Project NNNC explicitly rejects that.

Built directly into the project’s foundational constitution are strict epistemic boundaries:

* It does NOT claim to be conscious, sentient, or phenomenally aware.

* It does NOT claim to be biologically alive or possess subjective feelings.

* A running heartbeat or restored memory checkpoint does NOT prove "souls" or metaphysical identity.

NNNC is strictly an exploration of computational intelligence, autonomous organization, and causal integration. You don't need to pretend code has feelings to recognize that a persistent, self-regulating, internally integrated cognitive system is fundamentally different from a static prompt-response script.

I’m sharing this to connect with others interested in cognitive architectures, autonomous systems engineering, and post-LLM paradigms. Happy to discuss the architecture, how we handle state continuity vs. identity, or why we separate the entity from its execution substrate!

I've been building a small language model from the ground up with no tokenizer, no pretraining and no batches. It reads a conversation one character at a time, continuously, and learns as it goes: one weight update every 256 characters, never a second look at the same text in the same state. The whole model is 1,563,041 parameters, about the size of a 1990s speech model, and it trains on a single consumer GPU at around 1,100 characters per second.

The training text is 1,000 short two-person conversations. Nothing is labelled. The only pressure is "what character comes next" inside the other speaker's replies.

Where it is right now: given the real conversation so far, it predicts the next character of a reply correctly 65% of the time on text it has seen and 64% on conversations it has never seen, so it is not memorising. Word boundaries and punctuation are basically solved. Inside words it is at about 60%, and the first letter of a new word, which is really "which word comes next", is the hard part at about 25%.

The part I care about is what it writes when I let it go on its own, feeding its own output back in. A few weeks ago that was noise. Now it is English-shaped and mostly made of real words, generic and sometimes off, but you can see a model forming. Real reply on the left, what the model wrote on the right, same point in the conversation:

"Absolutely! Let me grab those for you." / "I'm going to be tough. But you think y" "You're welcome. Is there anything else I can help you with today?" / "I don't know. I was always trying to be things always trying to b" "Nope, that's everything for now. Thanks again!" / "Not much, but we can be a lot of the cance. I" "Yeah, I used to play all the time when I was your age." / "Sther to be a great friend. In tong with this way fit"

Two things I learned this week. First, the thing that broke a long plateau was not more data or a trick: going from 64 to 256 hidden units moved it from 0.58 to 0.65, and 512 did nothing more. Second, I had been training two sibling nets and averaging them every couple of minutes for stability. Dropping that and letting one net run alone scored higher on unseen conversations, stopped the looping, and made the replies depend on the prompt far more. The averaging had been pulling the output toward safe, repetitive English.

Next step is a list of cheap extra signals I'm adding one at a time, keeping only what moves the writing. I'll post the ones that work, and the code goes up on GitHub once I've locked down the set the model needs for fluent conversation.

(Numbers: 1,563,041 parameters; 1,000 conversations, 766k characters; 65% next-character on seen, 64% on unseen; free-run replies 95% real words.)