A medical study asked doctors to do something simple to understand and hard to do well: work out what might be wrong with a patient. GPT-4, the AI model behind ChatGPT, did it far better on its own than the doctors did with its help.
The study involved 50 doctors, who never saw the patients and worked only from written summaries of symptoms, test results and medical history. Some used normal medical resources, and others also had the AI model. The researchers wanted to know whether giving experts a stronger tool improves the work, a practical question facing every organisation using AI.
Doctors using normal resources scored 74 percent on the cases, and doctors using the AI model scored 76 percent. The study found that two-point difference was not strong enough to count as a reliable improvement. Tested on its own, GPT-4 scored 92 percent, which is why the result spread as ChatGPT beating doctors at diagnosing medical conditions.
Researchers also timed each case. Doctors with the AI model spent 519 seconds per case and those using normal resources spent 565, but the study did not show a clear speed improvement.
The study ran under controlled conditions and did not test full patient care, live conversations, physical exams, missing context or responsibility for a real medical decision. The finding is narrow, but many AI rollouts face the same problem of a strong tool and flat results.
A company gives skilled people access to AI and expects performance to rise. But the final decision still runs through the same habits, confidence and process. The company can spend money on AI licences, training and new workflows, then see little improvement if decisions do not change. If people treat AI as optional background input, usage can look high while performance stays flat.
The doctors did use the AI model. The paper says every doctor in the GPT-4 group tried it, based on chat logs. But they were not required to use it in a consistent way, and they received no specific training on how to work with it. The authors concluded that, in practice, access alone will not improve how doctors work towards a diagnosis.
