ESP32-AI: Tiny Language Model on $8 Chip Shakes IoT

A developer successfully ran a small language model on an ESP32-S3 chip. This chip is often found in smart plugs and low-cost sensing hardware, and costs less than a sandwich. The project, called ESP32-AI, uses a technique where much of the model is stored as a lookup table in flash memory. This means only a smaller part does the repeated calculations. The model has 28.9 million parameters, but only about 559,000 form the core that runs directly. The rest are in the lookup table. What's clever is that the firmware only reads the necessary rows from this table, about 450 bytes per token. The full model file is nearly 15 megabytes, which requires specific ESP32-S3 boards with 16MB of flash and 8MB of PSRAM. This is not a cloud AI killer or a general assistant like ChatGPT. It's trained for simple text generation, similar to TinyStories. This technology could be useful for specific tasks like classification or local text generation in devices like crop monitors or wearables. It shows the potential for more advanced AI at the very low end of on-device computing.