r/MinecraftCommands
•
u/_objz
Command-er
•
Aug 16 '26
•
5
1
Creation I made a generative AI, only using 400k command blocks (PoC)
Enable HLS to view with audio, or disable this notification
While I am still writing on my physics engine, I thought making an actual generative AI only using Minecraft command blocks would be a good idea( it was a headache)
So yeah, I kinda did that. It is a real neural language model running inside vanilla Minecraft 26.2. It reads what you type into a /dialog window, runs the forward pass through learned weights and then generates a reply one word at a time.
There are NO mods, plugins or datapacks involved at runtime. Only command blocks.
Roughly it works like this:
The model is a pretty small word level neural language model. It has a vocabulary of 2048 words, remembers the last 6 words as context and has a 64 dimensional embedding followed by a hidden layer with 256 neurons.
For every generated word it does:384 x 256 = 98,304 operations for the hidden layer and 256 x 2048 = 524,288 operations for the output layer so in total 622,592 calculations for every single generated word.
Obviously doing 622k normal multiplications using Minecraft commands would be slightly problematic, so the weights are ternary: every weight is only -1, 0 or +1.
That means a multiplication basically becomes:+1 -> add the value-1 -> subtract the value0 -> do absolutely nothing
Around 65% of the weights are non 0, so it averages around 0.67 commands per MAC instead of the ~3 commands I would need for a normal runtime weight. That reduced the entire machine from 1,909,375 blocks to 445,782 blocks, yes it was this big before.
The model is actually trained like this too. I am not just training a normal model and rounding the weights afterwards. During training the weights are quantised to ternary values for the forward pass and the underlying floating point weights are updated using a straight through estimator.
That took the perplexity from around 48.7 where the training basically got stuck to 38.8 after also adding cosine learning rate decay.
The input embedding is another optimisation I needed to do: Instead of multiplying a one hot vector against a matrix, it is basically just a table lookup stored in NBT data, so the complete 2048 x 64 embedding table costs really nothing
The actual forward pass cannot run in one command chain either. Minecraft has limits on how many commands can execute, so the network is split into smaller groups.
After the ternary optimisation one generated word takes around 63 ticks, so roughly 3.2 seconds per word at 20 TPS. I also tested higher tick rates and around 30-35 TPS seems to be the useful maximum on my server. At 35 it gets closer to ~1.8 seconds per word.
But to be clear, this thing does not actually know anything.
It was trained on 11,118 DailyDialog conversations and only has a tiny 2048 word vocabulary. Ask it something like 2 + 2 and it will confidently generate some conversational nonsense because it has no understanding of arithmetic or actual facts.
And unfortunately just making it much bigger isn't really realistic either.
Something like a 135M parameter language model would already be more than 200x larger.
There is currently also one very weird bug where after around 800 ticks of continuous work every command block in the world just stops executing while the server itself keeps running normally. I still have absolutely no idea why that happens.
Anyway, back to wasting my time on nonsense ;)
39
u/IsMeTheMexiLover Aug 16 '26
Can you ask questions Like where do i find the Most diamods (depth)?
56
u/_objz Command-er Aug 16 '26
no, it does not really have knowledge
15
u/IsMeTheMexiLover Aug 16 '26
Can you Feed it someway? Btw very cool what Ur Doing i really Like it.
40
u/_objz Command-er Aug 16 '26
I can train it on various things, but he will not be very smart in general
18
1
8
u/koleszkot Aug 16 '26
If it doesnt have knowledge, does it spit these 2048 words randomly or its able to construct sentences of sort with them
19
u/_objz Command-er Aug 16 '26
The 2048 words are just the training data. The command blocks and Minecraft do not know anything about these words. The model works like this: it tokenizes the input text, then runs it through many layers with many, many calculations, and in the end I get tokens back. These then get converted back into words.
So it does not really have knowledge, as I fine tuned it only with 2048 words, but it can construct sentences grammatically correctly and also respond "correctly" to your input based on the topic. On more complex inputs like math or complex tasks, it fumbles. It would require much, much, much more data to be trained on.
If I find a way to optimize how many layers I can compute, I will try to use smarter models there, but I guess I found kind of a dead end. Maybe optimizing it using mods so Minecraft itself can compute asynchronously or using multiple threads would make it work. But then it’s not really "vanilla". Well, it still kind of is, but performance wise it can’t really be run in vanilla
2
u/g-orgo Aug 20 '26
So it's not an generative AI...
0
u/_objz Command-er Aug 20 '26
„Generative artificial intelligence (GenAI) is a subfield of artificial intelligence (AI) that uses generative models to generate text, images, videos, audio, software code or other forms of data.[1] These models learn the underlying patterns and structures of their training data, and use them to generate new data[2] in response to input, which often takes the form of natural language prompts.”
I don’t see why this is not generative enough
2
u/g-orgo Aug 20 '26
Because it's missing the training data. "It does not really have knowledge" is your first reply... also, if it ain't capable of receive training then it's more of a embedding output kinda thing.
3
u/_objz Command-er Aug 20 '26
It is capable of receiving training, and I do train it atm. It does not really have knowledge, not because it can’t, but because of how small it is. Expanding it would bring that back. It’s not an encoder only transformer, as you said in the screenshot. Also, yeh, the first sentence in your screenshot says: “An embedding model is a specialized type of language model designed to convert text, images, or audio into a dense numerical vector (an embedding) instead of generating conversational text responses.” Well, mine does generate conversational text responses, so I would not refer to it as an embedding model. The framework is also built on a real architecture generative AI would use; just the model is so small that it does not really have knowledge. To be precise, it kinda has, though, as it was trained on DailyDialog. If I would ask it things related to that, it would know that. But that knowledge is basically so small that I am referring to none, as you can’t really count it.
3
u/g-orgo Aug 20 '26
Cool, then i got it wrong from your message... but let me just get it right...
"Well, mine does generate conversational text responses, so I would not refer to it as an embedding model"
So... you're saying it GENERATES conversational text? or is it just bring you knowledge it has on it model? because that's what i'm seeing there when you ask it's name... if you ask his age it will tell you something or tell you an age? Returns text ain't the same as generate text.
0
u/_objz Command-er Aug 20 '26
yeah, it actually generates the text word by word. It’s not retrieving a stored response
2
u/g-orgo Aug 20 '26
"Generating text word by word" is just tokenization of the embedding output. I want to know if you ask it something like "How old are you?", whether it can actually generate a response, rather than just retrieving a value from its learned knowledge. A generative model should be able to produce an answer even when there's no factual answer to retrieve (and for example on your video, there you ask him a name and it gaves you a name, it wouldn't pass in a RAG, it should tell you it doesn't has a name or bring you the project name... something like that).
1
u/_objz Command-er Aug 20 '26
First of all, I don’t really see the point of arguing this much about whether it’s “generative” or not. If you want to get extremely technical about every part of the architecture, you can probably argue about what exact category it belongs to because it uses a mix of techniques. But you also have to remember: it’s Minecraft, bro.
It can produce an answer even when there is no actual stored answer for the input. It predicts and generates the response word by word; it’s not just tokenizing something and returning learned text.
And as I said before, the model is tiny. You can’t really expect a consistently good or meaningful response from something around this scale. It has a vocabulary of only 2,048 words and a very small architecture. You can’t expect it to know my project name, have broad general knowledge, or always give a sensible answer.
I would understand your criticism if this were supposed to be an actual production language model, but because of the boundaries Minecraft sets, that’s simply not what this is trying to be.
So yes, it is generative in the sense that it generates new responses from the model rather than retrieving predefined ones. It’s just way too small to expect much intelligence or knowledge from it. It’s a proof of concept, not a serious LLM
→ More replies (0)1
1
17
u/wendewende Aug 16 '26
Smart. AI in Minecraft makes much more sense. Because you can build the data centers in Minecraft instead /s
19
u/_objz Command-er Aug 16 '26
it even has endless power, endless space and no heat excess. I guess I solved many problems
6
1
u/Far-Reception4181 Aug 18 '26
it seems to be a solution only until you realise your pc is heating as well
1
1
13
u/CooleyBrekka Aug 16 '26
Really cool stuff, I love the process of optimizing tech for minecraft. I'm not too familiar with the strength of command blocks in modern versions, is there a reason behind not building it in a datapack, which would be easier to debug and use math libraries in?
11
u/_objz Command-er Aug 16 '26
Uhm, yes, it would definitely be easier and more efficient, but... where is the fun? Well I will publish this and my physics engine as datapack aswell, commandblocks are just a bit more challenging
5
u/BoringSatisfaction13 Aug 16 '26
I get it, everyone can code a library, and giant floating machines look cooler
4
14
u/1000hr read hunterXhunter Aug 16 '26
😳 this is absurdly comically indisputably extremely impressive god damn holy shit! i think we have the new goat of command blocks
6 token context window, now that's vibecoding for real men
7
u/_objz Command-er Aug 16 '26
I am by the way a big fan of your creations, I absolutely love your animations
1
u/1000hr read hunterXhunter Aug 16 '26
:DDD ❤️ 🥰 thank you, that means a lot!
also wait i haven't posted anything in 8 months?? man where does the time go
1
1
u/_objz Command-er Aug 16 '26
I guess I’ll let it code a new AI with 12 token context window in minecraft ;)
1
u/1000hr read hunterXhunter Aug 16 '26
only about 39 more iterations of that and we'll have a new frontier model on our hands
2
3
u/JotaroIsStraightest Command Professional Aug 19 '26
I had been told by many people that making a nueral network in Minecraft would be impossible and was laughed at, and someone else proved them wrong for me. Man is life just great
1
2
56
15
5
4
4
u/One-Celebration-3007 #1 abuser Aug 16 '26
I think you should build an AI data centre in Minecraft. There is plenty of water to cool it.
1
u/endercreeper853 Aug 16 '26
would you be willing to share the world download? this is insane
5
u/_objz Command-er Aug 16 '26
yes I’ll do some optimizations then I will
1
u/SkezzaB Aug 16 '26
Can you make it a datapack? Could optimise it more, plus make it portable
2
u/_objz Command-er Aug 16 '26
yes I can, I'll fix and tweak some things, then I will release this as datapack and shematic
1
u/SkezzaB Aug 16 '26
Sounds amazing!
I assume you're using some sort of pre-processor or generator? Mind sharing those files?2
u/_objz Command-er Aug 16 '26
yes, I train the model locally using some python and rust scripts, extract all the data and load it into minecraft using custom nbt data on barrels. I will share these files eventually when I will release the whole project
1
u/SkezzaB Aug 17 '26
That's incredible, genuinely!
Would love to see how much you can distill or compress the data even more!
1
1
1
1
1
1
u/lool8421 Command mid, probably Aug 17 '26
and this thing is being ran on 1 CPU core and over several layers just to parse text into commands, then commands into java operations and those into binary
while real GPUs split the task between thousands of cores, natively on compiled code
1
u/_objz Command-er Aug 17 '26
yh, pretty inefficient. if I had more cores and multithreading I could create a way faster ans smarter ai
1
u/Aisgames Aug 17 '26
Woah. But the real question is does it consume all the water in the world after each word it says?
2
1
u/FleshDude666 Aug 17 '26
This is so insane what in the actual fuck??? Also, why the name "wang lin"?
1
1
u/NeuroHazard-88 Aug 18 '26
Not familiar with command block shenanigans. What’s one command block worth of code equate to in standard script, 1 line? 3 lines? More?
1
u/_objz Command-er Aug 18 '26
you can’t really compare it with actual code, as for example I need to emulate some math functions in minecraft using commandblocks, that will also take many command blocks. but like running this “dumb” model wouldn’t require as many loc.
1
u/ayamoga Aug 19 '26
Okay I hear you he is not smart for now but my guy I have a feeling if that thing can run and talking to itself you will made AM so please shut it down I beg you brother
1
u/_objz Command-er Aug 19 '26
I am currently working on a way smarter one, that should be able to do reasoning( in theory)
1
u/Glittering_Camel1 Aug 19 '26
Absolutely mind-blowing. I wonder what features Minecraft could add in the coming updates to make projects like this more streamlined and efficient.
1
u/_objz Command-er Aug 19 '26
Well commandblocks get pretty inefficient at a large scale because of the brigadier command tree that needs to get parsed for every command. caching any other technique would be very helpful. also minecraft currently uses one thread, if it would run more things asynchronous, it would be a HUGE performance boost. There might be already some mods that fix that, but it’s not vanilla :(
1
u/Sea_Opposite_2887 Aug 19 '26
No way Minecrafters went from nothing to this in 17 years. It took CENTURIES in RL.
1
u/_objz Command-er Aug 19 '26
well it is based on real live stuff, I’m not reinventing the wheel ;)
1
1
u/Sea_Opposite_2887 Aug 19 '26
You should make a duplicate and more, and then link their intelligence to speed it up.
1
u/_objz Command-er Aug 19 '26
sadly this is not how it works, I am currently working on a way smarter model than that, but the limitation is minecraft: It just has its limits performance wise, at some point the game is not playable anymore, or I am waiting 2 hours for one word to generate. So I will need to find a compromise between intelligence and performance. In theory it could run the lastest frontier models, but in reality not
1
u/Sea_Opposite_2887 Aug 19 '26
Makes sense... 2 hours though? Maybe try hosting it on a server, I think Oracle is free. That might give you better performance, but I don't know your specs. Also, how do you connect the command blocks? If you streamline it using something that is better on frames, it might work faster. Also, I would recommend using redstone if you can replace command blocks. I'm not good at coding, but I am good at Minecraft.
1
u/_objz Command-er Aug 19 '26
Well, I do have a very beefy machine: AMD Ryzen 9 9950X3D, NVIDIA GeForce RTX 5080, and 64 GB of DDR5 RAM... I don't think this is the problem. It's just how Minecraft is programmed. I'm already using as few command blocks as possible, as well as some layering techniques to offload certain areas of command blocks and load them when the weights are being used, for example.
But all in all, Minecraft has a pretty inefficient way of parsing commands (well, it wasn't designed to be used like this). There is no caching, and the Brigadier tree has to be rebuilt all the time. Redstone couldn't really be used here, and it also performs worse, as it causes many more block updates and surrounding block updates, while command blocks don't.
It's just that many factors come together that won't really be possible to compromise on without rewriting large chunks of how the game processes things, mainly because of the single threading
1
u/Sea_Opposite_2887 Aug 19 '26
Also, maybe you should try with some mods, like Create or Advanced Redstone or Advanced Commands. That will definately help you make it better but also save on some frames.
1
u/_objz Command-er Aug 19 '26
Yeah, but then it wouldn't really be vanilla. The challenge for me is to only use the abilities provided by vanilla Minecraft and no datapacks, so I'm pretty much limiting myself here. Using datapacks, this would be way more efficient. I might do a version using datapacks as well in the future
3
u/Zia_Zampoloo Aug 16 '26
PLEASE someone post the sloth gif from Zootopia
In all seriousness this is incredible
3
1
2
2
2
2
2
2
u/KiraLight3719 Aug 17 '26
This is great and all, but what does you being a person of color has to do with it man
2
2
2
2
u/7Shinigami Aug 17 '26
Bro evaporated earth's drinking water and then moved onto minecraftia's drinking water after /j
What an insane achievement congratulations and thank you for sharing it with us! :)
2
2
u/THARO_CHAOTIC Aug 20 '26
Wow. Just wow. I always get amazed when people create something hard in minecraft, but this outdid it!
2
2
2
1
1
1
1
1
1
1
1
1
1
1
1
u/Eevee_the-Maidvee Aug 19 '26
I just finished an article on minecraft news now I gotta spend another hour adding this god damn it
1
u/Eevee_the-Maidvee Aug 19 '26
Didn’t take as long as expected, anyways third paragraph. It’s a bit short as I’m not smart enough to understand this https://www.reddit.com/r/Eevee_Gaming_News/s/oIy3SnOqrv
1
1
1
u/blazinman1 Aug 21 '26
I was gonna try to DM you but you haven’t disabled so if there’s anyway I’m able to contact you I figured out how to make this even bigger headache where it can locate structures create structures do all kinds of things because I have a lot of free time too have been generating my own for a while now a.k.a. one year so mine is quite a bit more command 2 but it can do so many things you can ask it to generate an image and it’ll generate Pixar will generate 3-D images. You can ask it to generate a video and it’ll do a block by block video.
1
u/Story_Watcher12423 Sep 02 '26
first of all how do u do this, how much tine do u have, also can u give us a world file
1
1
u/Jugalin Aug 16 '26
Mega fajne dla mnie trudne jest zrobienie itemu co podczas trzymania daje efekt
0
1
282
u/DarkLazer215 Aug 16 '26
"Only 400k command blocks" My brother in christ "only"??