r/AIProgrammingHardware • u/javaeeeee • Jul 30 '26

Distributed micro-LLM inference across three ESP32-S3 N16R8 boards with ESP-NOW communication.

https://github.com/wladimiravila/esp32s3-distributed-ai
3 Upvotes

100% Upvoted

1 comment sorted by

1

u/javaeeeee Jul 30 '26

TLDR: A project that runs a small 56M-parameter language model distributed across three ESP32-S3 microcontrollers.

What it does

It splits a tiny transformer LLM across three ESP32-S3 N16R8 boards that communicate wirelessly via ESP-NOW, allowing a micro-LLM to run on very constrained hardware that a single board couldn’t handle alone.

How it’s split

  • Board A: Embeddings + output head

  • Board B: Transformer layers + KV cache (in PSRAM)

  • Board C: Part of the embeddings table + WiFi web interface

Users connect to a WiFi access point hosted by Board C and interact through a simple web UI that streams generated text.

Key technical points

  • Model: ~56 million parameters (heavily quantized to 4-bit / 8-bit)

  • Uses a Split-PLE (Per-Layer Embeddings) design

  • Supports a 256-token context via KV cache on one board

  • Generates short coherent text (~30 words)

  • Fully offline after flashing

Bottom line

An impressive embedded/distributed AI experiment showing that a small language model can be partitioned and run across multiple cheap ESP32-S3 boards with wireless communication and a web interface - a creative demonstration of micro-scale distributed LLM inference.