LLMs Need "World Model" for Real-World AI: AMI Labs

Pascal Feng, co-founder of AMI Labs, says Large Language Models, or LLMs, understand the world through human texts. He explained the need for a "world model" for AI agents to operate effectively in the real world. This keynote speech was given at the ICML Conference in Korea. Feng states that AI is shifting its focus to operating in the real physical world. A world model refers to AI that understands the world beyond just text. He compared this to a soccer player who understands spatial relationships, physical actions, and even emotions. AMI Labs was founded by New York University professor Jan LeCun. The company has already attracted one billion dollars in seed investment this year. LeCun believes LLMs alone have limitations for real-world AI. Feng emphasized that LLMs learn from texts, which is an indirect way of understanding the real world. He added that LLMs focus on fluent dialogue, not on understanding physical causal relationships. This makes them slower and more expensive than humans for world understanding. He also warned that "hallucinations" in LLMs, harmless in text, could cause collisions if they occur in robots. Current top AI models perform worse than humans in understanding the physical world. Humans scored 64.7% in a physical inference benchmark, while the best AI model only reached 41.2%. This shift in focus could lead to more capable and safer AI systems interacting with our physical environment.