Clinical AI vs. Chatbots: General-Purpose Models Win Benchmarks

General-purpose chatbots are outperforming specialized clinical AI tools in ground-level performance tests. This is a "Validation Inversion" where tools specifically built for healthcare are losing to chatbots that regulators previously warned against using. A study in Nature Medicine showed general-purpose models won against specialized clinical AI tools on real-world questions physicians ask. These general-purpose models outperformed purpose-built clinical tools, even those that cost less. What's interesting is that general-purpose large language models, like GPT-4, are trained on vast amounts of medical literature, clinical guidelines, and case documentation. This dwarfs the datasets most specialized clinical AI vendors use. Another study found that ChatGPT with GPT-4 was more accurate than emergency department resident physicians in diagnosing internal medicine cases. The FDA's updated guidance on Clinical Decision Support software creates a challenge. It may incentivize vendors to keep their tools just below the device threshold, avoiding rigorous validation that could reveal performance gaps against general-purpose alternatives. The bottom line: This raises questions for anyone in clinical trials using AI-assisted tools about how these tools compare to general-purpose large language models.