Language Models' "Amnesia": Why AI Needs a Nap

Language models, despite their advanced capabilities, are experiencing what researchers call "anterograde amnesia." This means they don't learn anything new after their initial training. Here's the thing: once a model is trained, its knowledge is essentially fixed. It can process information up to its data cutoff but remains unaware of more recent events. While new facts can be temporarily fed into its context window, this information is lost once the session ends. Researchers Behrouz, Hashemi, and Mirrokni from Google and Cornell describe this as similar to a patient with anterograde amnesia, who retains old memories but cannot form new ones. The models have deep past knowledge and an immediate present, but nothing connects them for long-term learning. The paper suggests that, like humans, these models might be missing a crucial step that biology figured out: sleep. This concept challenges how we approach ongoing learning and memory in artificial intelligence.