Sanskrit GPT: Unicode Challenges & Grapheme Fix

A developer built a small GPT model that speaks Sanskrit over a weekend. The goal was to predict the next syllable. Here's the thing: while building a character-level model, the developer discovered that a "character" in English is straightforward, but in Sanskrit, it's complex. What appears as one syllable in Sanskrit can be multiple Unicode code points stacked together. For example, the syllable "कि" is two code points, and "श्री" is four. This means a standard way of defining a vocabulary by splitting text into individual code points creates "orthographic shrapnel" for Sanskrit. The model would then have to learn Devanagari spelling before it could learn the language itself, leading to nonsensical outputs like floating vowel signs. The fix involves splitting text into "grapheme clusters" instead. These are the chunks humans perceive as characters. Using this method, the grapheme tokenizer produces 44% shorter sequences, giving the model more context and ensuring every token is a valid, on-screen syllable. The bottom line is that choosing the right "unit" for the model's vocabulary was the most crucial decision in this project, preventing invalid output from being generated.