UNIST AI Cuts LLM Size by 63% with GMoE Tech

A new AI technique can cut the size of large language models by 63% while keeping their performance. This new architecture is called GMoE, or Global Mixture of Experts. It allows multiple neural network layers to share AI modules. Here's the thing: existing models often have experts in each layer, leading to a massive increase in parameters as models get deeper. GMoE solves this by using "shared experts" across all layers and "dedicated experts" for each layer. For example, it can build an entire structure with 1,100 experts, compared to 100,000 in previous methods. What's interesting is that in tests, a mid-size model saw its parameters drop from 549 million to 204 million. Its average accuracy was 39.51%, nearly identical to an existing model almost two and a half times larger. This technology also helps distribute inputs more evenly among experts, using more pathways and reducing concentration on a single one. The bottom line is this could make advanced AI more efficient and accessible for everyone.