A developer has successfully run a 28.9 million parameter large language model on an ESP32 microcontroller costing $8. This demonstrates the feasibility of deploying LLMs on extremely low-cost, low-power hardware, potentially enabling edge AI applications.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
Developer slvDev successfully deployed a 28.9M-parameter language model on the ESP32-S3 microcontroller, using Google's Per-Layer Embeddings technique to store most parameters in flash memory, achieving a generation speed of approximately 9.5 tokens/s. This result increases the upper limit of model parameters on similar chips by about 100 times.
Microcontrollers typically have only 512KB of SRAM, and traditional methods require the entire model to reside in SRAM, limiting model size. slvDev's solution stores most parameters (25M) in flash, keeping only the inference core in SRAM. Each token generation reads only about 450 bytes of embedding vectors from flash, rather than loading the entire model.
This design draws on the Per-Layer Embeddings concept from Google's Gemma 3n and Gemma 4, but is applied for the first time to a microcontroller rather than a phone or GPU. The developer states that, to his knowledge, no one has attempted this on such a small chip before.
The model achieves approximately 9.5 tokens/s generation speed on the ESP32-S3, with a pure compute speed of 9.7 tokens/s. All inference is performed locally on the device, requiring no network connection.
However, the model's capabilities are limited by its small inference core. It is trained only on the TinyStories dataset and can generate short, coherent stories, but cannot answer questions, write code, or execute instructions. The developer explicitly states that this limitation stems from the model's small inference part; the memory optimization does not change its capability ceiling.
Previously, the largest language model running on similar chips had only 260K parameters, while the new model's parameter scale has increased by about 100 times. The developer has publicly released complete training, quantization, and deployment code, along with detailed experimental results (including the correction process for parameter count errors) for community reproduction and verification.
The project code is hosted on GitHub, including firmware, training scripts, and measurement data. The developer specifically notes that early parameter counts contained errors that inflated the numbers; corrected results have been updated in the repository.
This article is based on the project documentation and experimental results publicly released by developer slvDev on GitHub. All technical details and performance data come from the project's README and RESULTS.md files. The developer proactively disclosed parameter count errors and the correction process, increasing credibility. However, the model's capability limitations (only generating simple stories) are by design, not a performance flaw.
By storing embedding tables in flash, a 28.9M-parameter language model can run on an $8 microcontroller with an inference speed of about 9.5 tokens/s. This method increases the parameter scale of LLMs on microcontrollers by two orders of magnitude, but the model's capabilities are limited by the small inference core, making it suitable only for simple text generation tasks.
Primary report
Primary source
Also covered by 1 sources