Google AI: Sign Language to Text for Web Search

Google has developed a new sign-language translation model that converts complex body movements into text. This technology, called sign-language-to-text or SL2T, is now moving into consumer devices, starting with American Sign Language on Pixel 11. Users can sign to search the web, write messages, or interact with Google’s Gemini. In Live Transcribe, people can also sign responses during conversations. This system was trained on over 100,000 hours of data across more than 50 sign languages. What's interesting is that sign language translation is both a computer vision and a language translation problem. The system uses body landmarks, like geometric coordinates of hands, face, and body, instead of raw video, to protect privacy. This approach reduces sensitive visual information leaving the device. The model translates sequences of these landmarks directly into text, avoiding intermediate "glosses" to better capture spatial relationships and non-manual signals. This innovation could significantly enhance communication and web access for sign language users.