What The Science Can Say About Microsoft’s AI And Doctors

🔍 Read the full analysis: What The Science Can Say About Microsoft’s AI And Doctors on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get smart everyday buys delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic system scored higher than generalist physicians in a study of historical medical cases published Oct. 22 in Science, but the research tested a text-based simulation rather than care delivered to real patients. The company reported 83.2% accuracy on 304 complex cases, compared with 36.1% for physicians using textbooks and the internet and 22.3% without those resources.

The paper, Sequential diagnosis with language models, also reports results from 325 emergency department cases: MAI-DxO scored 80.3%, compared with 46.8% for physicians. The results are Microsoft’s, and all 15 paper authors are listed with Microsoft affiliations, including Microsoft AI chief executive Mustafa Suleyman. Publication in a peer-reviewed journal means the paper underwent review; it does not mean independent researchers have reproduced its findings.

The study used Microsoft’s Sequential Diagnosis Benchmark. Each case began with a short summary, and the system or physician could request further information, such as a test result, from a model holding the case file. The benchmark assessed both the final diagnosis and the estimated cost of tests. MAI-DxO coordinates a language model acting as a panel of virtual physicians, which proposes diagnoses, requests information and weighs further testing over successive rounds.

The comparison was not between the system and specialist teams. In the earlier preprint’s description, participating doctors were 21 U.S. and UK generalists: 17 in primary care and four hospital generalists. They worked individually. The final paper’s reported physician results include an open-book group, but the study did not include specialists or teams. The cases were selected for difficulty and came from published records, not from patients being examined in a clinical setting.

What Diagnostic Accuracy Leaves Out

The findings offer evidence that language-model systems can perform well on a particular kind of diagnostic task: gathering information from a case record and identifying a final diagnosis. That is relevant to hospitals evaluating clinical AI, but it is not yet evidence that MAI-DxO improves patient outcomes, reduces delays or lowers costs in routine care. Those questions require tests in actual clinical workflows.

The reported comparison also needs to be read alongside the study design. A system combining several virtual agents was compared with individual generalists, while the difficult cases would often be referred to specialists or discussed by clinical teams. The benchmark’s chosen cases also do not represent the full range of patients seen in practice. As a result, the accuracy gap cannot be treated as a direct measure of how the system would compare with a hospital’s usual diagnostic process.

For patients and health systems, the cost estimates are not a demonstrated savings figure. The study priced tests using billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit charge. The paper’s authors describe those calculations as approximations. Actual costs and clinical value would depend on how the tool is used, what tests it prompts, and whether its advice changes care.

How the Benchmark Worked

Unlike a conventional medical exam in which every case detail is available at once, SDBench presents a short initial description and lets the participant request additional information. Each request has a price in the benchmark. This design is intended to test not just whether a diagnosis is eventually reached, but how the system gathers information along the way.

Some methodological details in the supplied account come from the 2025 preprint, which preceded the final Science paper; the final article’s full text is paywalled, so those details may not all match the published version. In the preprint, a separate model scored diagnoses against the published answer, and a physician review found agreement that was good but not perfect. When a requested result was absent from a case record, a gatekeeper model generated a synthetic result rather than revealing that it was unavailable.

The preprint also described a limited physician comparison using the 56 most recent held-out cases, while the AI was tested on all 304. Microsoft’s final Science results report the physician averages as 22.3% without references and 36.1% with textbooks and internet access. The distinction between the preprint and the published paper matters when interpreting the study’s methods and numbers.

“The test costs are “first-order approximations.””

— Microsoft’s paper authors, as described in the study material

Limits Before Clinical Use

The study does not show whether MAI-DxO can diagnose patients accurately during live care, when histories may be incomplete, examinations and imaging matter, and clinicians must weigh harms as well as possible diagnoses. It also does not establish whether the system works as well with specialists, multidisciplinary teams, or the range of ordinary cases seen in clinics and emergency departments.

The source account says the benchmark’s case sets contain no healthy patients and no harmless explanations. The authors noted in the preprint that this prevented them from measuring false alarms. That leaves open how often a system tuned to find serious or unusual conditions might recommend unnecessary tests or raise concerns in cases that have benign explanations.

It is also unclear how well the reported performance would hold up when tested by researchers outside Microsoft. Peer review evaluates a paper’s methods and reporting, but the supplied material does not identify an independent replication. The company’s accuracy and cost figures should be understood as results from this study, not established performance guarantees.

Testing With Patients at Beth Israel

The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the supplied source, consenting patients will interact with the system alongside their normal care. That setting could provide evidence about how the tool behaves in clinical use, but the available information does not specify the study’s start date, size, outcome measures or publication timeline.

Results from that work will be needed to judge whether the benchmark performance translates into useful and safe support for clinicians. Until then, the Science paper establishes what MAI-DxO scored on the tested case records—not how it performs as a diagnostic service for patients.

Key Questions

What is MAI-DxO?

MAI-DxO is Microsoft’s diagnostic AI system. It uses a language model in a multi-agent setup to propose diagnoses and request further information during a case.

What did the Science study find?

Microsoft reported 83.2% accuracy for MAI-DxO on 304 complex cases, versus 36.1% for generalists using textbooks and the internet and 22.3% without those resources. On 325 emergency department cases, it reported 80.3% for the system and 46.8% for physicians.

Does the study show that MAI-DxO is better than doctors in hospitals?

No. It tested historical case records in a text-based simulation, comparing the system with generalists working individually. It did not test real-patient outcomes or compare MAI-DxO with specialist teams.

Have independent researchers confirmed the results?

The supplied material does not report an independent replication. The paper was peer-reviewed, but that is different from another research group reproducing the results.

What is the next planned test?

A prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients are expected to interact with the system alongside their usual care. Further details, including its timeline and measures, were not provided in the source material.

Source: ThorstenMeyerAI.com