OpenAI Astra: Shifting Benchmarks & Performance Concerns

OpenAI has been changing evaluation benchmarks for its GPT-6 Astra model since its blog post announcement on September 3rd. In some cases, the updated numbers show Astra performing better, while metrics for rival models appear worse. This happened during an unusual rollout of the blog post. OpenAI initially struggled to get the post widely viewable online, with links returning error messages. The company stated it retracted the blog shortly after 2 PM for reasons it could not disclose, but said these were unrelated to benchmark performance figures. Upon republishing, the blog featured different evaluation metrics that seemed to favor Astra, and some figures have continued to change even since then. For example, Astra's reported hallucination rate was initially 4.2%. This was later halved to 2% in an updated version of the blog. An OpenAI spokesperson stated they are making fixes to ensure numbers represent their best estimate of model performance. This situation highlights the challenges of measuring large language model performance and concerns about potential manipulation of specs.