AI Fails Scientific Task: Frontier Models at 3%

A new benchmark shows that advanced AI models struggle significantly with a key scientific task. Every large language model tested could only reconstruct a research paper's core finding from its reference list alone between three and fifteen percent of the time. This benchmark, called Reconstruction, was posted to arXiv on August 17th. It tests seven frontier models using 643 papers across six scientific areas. The models were given only the reference list of a paper, with no full text, author names, or other identifying information. The low scores highlight how far today's AI is from forming truly novel hypotheses. What's interesting is that even a multi-agent system, which used competing hypotheses, only reached between twenty-three and forty-two percent accuracy. The benchmark's design prevents models from simply pulling answers from their training data. It uses a temporal citation cutoff, anonymous reference IDs, and frozen bibliographies to ensure a clean test. The scoring mechanism uses an independent LLM judge to match model-proposed hypotheses against actual findings. The bottom line is that no single model performed dramatically better than the others, showing a consistent failure rate across different advanced AI systems. This suggests that the issue isn't just about prompting or model size, but a fundamental limitation in genuine scientific discovery.