Sanskrit GPT: How Grapheme Clusters Solved "Orthographic Shrapnel

A developer built a small GPT model over a weekend to predict Sanskrit syllables. The goal was to understand transformers by building one from scratch. The key challenge was how Sanskrit "characters" are represented. In English, predicting the next character is straightforward. But in Sanskrit, what looks like one syllable can be made of multiple Unicode code points. This means a standard character-level approach, which works for English, produced "orthographic shrapnel" for Sanskrit. The model would generate nonsense because it was trying to learn from bare consonants and floating vowel signs. The solution was to split the text into "grapheme clusters" instead. These are the chunks humans perceive as single characters. Using this method, the model's vocabulary became larger, but the sequences it processed were 44% shorter. This allowed the model to see more real context and ensured every token it generated was a valid syllable. This choice of unit was the most important decision in the project, making invalid output impossible. This shows how crucial foundational data processing is for building effective language models.