Language Model Training: Raw Learning Process Revealed

A new project shows how language models learn, making every calculation visible. It focuses on a small scenario where you can see incorrect answers before learning and correct ones afterward. The project uses a single command and requires Node.js 18.19 or higher, with no other dependencies. Initially, after random setup, the model gives "wrong" answers to questions like "can human read?". But after training, which includes pre-training and adaptive SFT, the same model provides "correct" answers such as "human can read." The final version of the code simplifies the learning process into one clear pipeline, from text tokenization to backpropagation. Every number involved in learning is treated as a "Value" object, and a neuron is literally an object, making its function transparent. This approach offers a detailed, step-by-step view of how a language model learns, which can help in understanding the underlying mechanisms without complex framework abstractions.