🔍 Read the full analysis: The Referee Shortage: AI’s Growing Need For Human Oversight on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A recent analysis from ThorstenMeyerAI.com describes a growing gap between AI’s ability to produce work and people’s capacity to check it. Its examples span mathematics, software and legal workflows, but the figures come from sources with different methods, and some are provided by vendors of review tools. The scale and long-term effects of the gap remain uncertain.
A report published by ThorstenMeyerAI.com argues that AI is producing mathematical, software and contract work faster than people can verify it, making human review capacity a growing constraint. The examples include OpenAI’s publication of 722 mathematical manuscripts this week and software-industry figures showing more code changes alongside longer or absent reviews; the evidence comes from separate sources and does not establish one universal rate of AI error.
The report says OpenAI posed about 4,000 mathematics problems and generated 722 manuscripts grouped into 372 families. Some results were formally checked using Lean, a proof assistant. OpenAI cautioned that unformalized results could have issues. The report contrasts that volume with the careful examination by five leading mathematicians of an earlier result from the programme that was presented as a counterexample to an Erdős conjecture. It does not provide enough detail here to establish how much review each of the 722 manuscripts received.
In software, the report cites Faros AI data indicating teams merged 98% more pull requests during high-AI-adoption periods while review time rose 91%. LinearB, which analysed 8.1 million pull requests across 4,800 organisations, reported that AI-generated changes waited 4.6 times longer for review to start and were accepted 32.7% of the time, compared with 84.4% for human-written changes. A peer-reviewed 2026 study found 61% of AI-agent pull requests received no human review before being merged or closed.
The report also points to an OpenAI partnership with contract-software company Ironclad. It says GPT-6 Astra met an average of 55% of evaluation criteria across 11 contracting tasks, an improvement on its predecessor. The report interprets the remaining criteria as work requiring further checking. It does not give the full evaluation protocol or specify how the results translate to live contracts.
The referee shortage: AI made doing cheap and checking expensive
OpenAI’s model produced a maths result in about three hours of compute. Verifying one earlier result took five of the world’s leading mathematicians. That ratio is the next decade of work: producing is cheap and abundant; trusting is slow, human and scarce.
Some Lean-checked; OpenAI warns unformalized ones “could have issues.” Verification abundance, adjudication scarcity.
Faros AI. LinearB (8.1M PRs): AI changes wait 4.6× longer, accepted 32.7% vs 84.4%.
Real progress. Someone still has to find the other 45% before the work can be used.
Machine output arrives without reasoning a reviewer can interrogate. It looks locally clean and gives no clue where it’s wrong.
A prover confirms the proof proves its statement; tests confirm what tests check. Neither confirms it’s what was needed.
Contracts are signed, designs stamped, papers defended. Responsibility is institutional — you can’t hold a model to it.
of AI-agent pull requests got no human review at all (EASE 2026). Zero-review merges up 31.3% (Faros).
of reviewers deliberately deprioritise AI changes (LinearB). Good machine work waits behind bad.
OpenAI chose which maths families were significant. When referees can’t keep up, the producer’s filter becomes the review.
Budget review hours next to model spend.
Provers, types, tests, policy engines.
Experts only where consequences are high.
Who profits from generation pays for checking.
Keep some production human for learners.
The first automation question was which jobs AI would do. The better one is which jobs AI makes more necessary: the ones that check, adjudicate and take responsibility. Expect a referee premium — senior engineers, auditors, specialist lawyers, reviewing scientists become the binding constraint on how much AI output anyone can use.Accountability — standing behind a result — may be the most durable form of human work there is.
Human Review Becomes the Constraint
If AI makes drafting, coding and research cheaper, organisations may be able to produce more work than their reviewers can responsibly assess. The immediate concern is not simply that a model can make mistakes; it is that a larger volume of output can stretch review time, leave some work unchecked or encourage reviewers to sort submissions by suspicion rather than quality.
The report describes three possible consequences: rubber-stamping, deprioritising machine-generated work, and relying on the producer to decide what deserves attention. Its cited figures illustrate why those risks merit scrutiny, but do not prove that every organisation is experiencing them. Reviewers also provide something a technical test alone may not: a person or institution able to explain, correct and take responsibility for a decision.
That pressure could affect staffing and training. Experienced engineers, lawyers, scientists and auditors acquire judgement through doing and reviewing the underlying work. If AI tools replace too many entry-level tasks, future organisations could have fewer people prepared to check complex output. The report presents this as a risk, not an established outcome. Its broader point is that the capacity to judge and take responsibility may determine how much AI-generated work can be used.
As an affiliate, we earn on qualifying purchases.
Three Fields, Different Checks
The examples in the report concern different kinds of verification. In mathematics, formal systems such as Lean can establish that a proof follows from specified assumptions and reaches a specified conclusion. That does not by itself establish that the theorem addresses the right question or is significant. In software, automated tests check only the conditions they cover; a passing test suite does not guarantee that a change meets every user or security need.
Contract work brings a separate accountability problem. A draft must fit the relevant agreement, jurisdiction and approval process. The report says OpenAI and Ironclad trained GPT-6 Astra on contracting workflows, but an average score against evaluation criteria is not the same as proof that a contract is ready to sign. Across these fields, the report’s distinction is between checking a defined answer and deciding whether the task, assumptions and outcome are appropriate.
The comparison has limits: a mathematical manuscript, a code pull request and a contract are not interchangeable units of work, and the figures cited use different methods. Faros AI and LinearB sell or provide software-development analytics and review products, so their findings should be read with attention to their methods and commercial interests. The report itself cautions readers to treat several software figures carefully.
“”verification abundance, adjudication scarcity.””
— ThorstenMeyerAI.com report
As an affiliate, we earn on qualifying purchases.
How Much Review Is Missing?
The cited data does not establish how often AI-generated work contains errors, how severe those errors are, or whether the reported patterns apply across industries. The software figures come from differing datasets and definitions, and several cited sources have commercial interests in code-review tools. The report does not provide enough methodological detail to compare the statistics directly.
It also remains unclear how many of OpenAI’s 722 manuscripts were independently reviewed, how the 55% contracting evaluation was designed, and whether GPT-6 Astra’s results predict performance in real legal work. The report’s claims about reduced opportunities for junior workers and a future premium for expert reviewers are forecasts and interpretations, not measured outcomes established by the figures presented.
mathematical proof verification software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Tracking Review and Training
The next useful evidence will show not just how much AI-generated work is produced, but how it is reviewed and what happens after deployment. That includes transparent reporting on error rates, review time, unreviewed changes, correction costs and the consequences of failures. For mathematics and legal work, independent evaluation would help readers understand what a model’s formal or benchmark score does—and does not—show.
Employers and professional bodies will also need to watch whether junior workers still get opportunities to draft, code and build the expertise that later supports review. The report offers no timetable for a change in hiring or professional standards. For now, its central claim is a capacity warning: AI output is growing, while the supply of accountable human judgement may not grow at the same pace.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main development described in the report?
ThorstenMeyerAI.com argues that AI is producing work faster than people can check it, citing mathematics, software and contract workflows. The examples come from different sources and should not be treated as a single industry-wide measurement.
Did OpenAI say all 722 mathematical manuscripts were verified?
No. The report says some results were formally checked in Lean and quotes OpenAI warning that unformalized results could have issues. It does not establish that all manuscripts received independent expert review.
What did the software figures find?
The report cites Faros AI figures showing more pull requests merged alongside longer review time, LinearB data on longer waits and lower acceptance rates for AI-generated changes, and a 2026 study finding that 61% of AI-agent pull requests had no human review before closure or merging. Their methods and populations differ.
Does formal verification remove the need for human reviewers?
Not necessarily. Formal tools can check whether a result meets specified rules, but people may still need to decide whether the rules reflect the right question, whether the result is useful and who is accountable for acting on it.
What remains unknown?
The cited material does not establish the overall error rate or harm from AI-generated work, the share of manuscripts independently reviewed, or whether the reported software patterns apply broadly. The long-term effect on professional training and reviewer staffing is also uncertain.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
