🔍 Read the full analysis: Can ChatGPT’s Data Science Gains Become Lasting Consultant Skills? on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT with brief training scored higher than a control group on coding, statistics and prediction tasks. When tested afterward without the chatbot, the researchers found no evidence that the ChatGPT group had gained more technical knowledge.
A randomized trial involving consultants at Boston Consulting Group found that brief training and access to ChatGPT improved their performance on data-science tasks, but researchers found no evidence of a lasting advantage when participants later answered technical questions without the tool. Published October 26 in Nature Human Behaviour, the study bears on whether companies should count AI-assisted work as employee skill development.
The researchers compared consultants who received 15 to 20 minutes of ChatGPT training and access to GPT-4 with a control group trained to use Google and online resources including Stack Overflow and Khan Academy. Participants were asked to complete two of three 90-minute tasks: cleaning and combining data with Python, evaluating a statistical model, or building a model related to the predictability of international football matches.
Among the 487 consultants who submitted work on both assigned tasks, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. The study measured results against the average performance of 44 BCG data scientists who completed the tasks without AI. The consultants using ChatGPT still scored below that expert benchmark on average across all three tasks.
After the work session, participants answered technical questions on related topics without ChatGPT. The authors report that the ChatGPT group did not perform better than the control group on this follow-up assessment. They describe the task-time benefit as temporary upskilling: the tool helped participants do more while they had access to it, but the results did not show that they had learned to reproduce that performance independently.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
AI Help Is Not the Same as Training
The findings distinguish work enabled by AI from knowledge retained by the worker. For employers, that difference can affect how they evaluate training programs, assign technical work and check its quality. A consultant who can complete a coding or statistics task with a chatbot may be able to contribute to a project sooner, but the study does not support assuming that the experience alone has made that person independently proficient.
The paper’s results also show why performance gains should be assessed alongside the conditions under which they occurred. In this trial, ChatGPT improved task scores and, according to the authors, shortened average completion time for coding and prediction. Yet the researchers did not find a corresponding advantage when the tool was removed. Organizations using AI-assisted work may therefore need to distinguish between the output an employee produces with assistance and the skills they can demonstrate without it.
The study also reports a potential calibration concern: the ChatGPT group became more confident that the tool could solve problems it could not actually solve. The authors characterize this analysis as exploratory, so it should not be treated as a settled finding about all users. Still, it highlights a practical reason to retain review and verification when employees use generative AI on technical work.
Products Worth Considering
Dell UltraSharp U4025QW 40 5K2K WUHD Curved LED Monitor (21:9)
Canon IVY 2 Mini Photo Printer Bundle, Blush Pink
Dell Pro 14 Plus P1425 14-Inch Portable Monitor
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The experiment took place at BCG in late March and early April 2024 and was preregistered before data collection, on March 13. Researchers allocated 986 consultants equally between the treatment and control groups; 573 began the survey, and 487 submitted work on both tasks used in the main analysis. Participants had at least a bachelor’s degree, and the authors say most reported no coding experience.
The tasks were developed by BCG data scientists with OpenAI’s Economic Impacts Research team. The researchers designed them so ChatGPT could offer useful assistance but could not correctly answer every question on its own. The comparison with 44 BCG data scientists provided an expert reference point, while the later tool-free questions tested whether participants showed a detectable technical knowledge gain after using ChatGPT.
The study’s title, “Generative AI and the temporary upskilling of knowledge workers,” reflects its central distinction. The authors compare AI assistance to an exoskeleton: it can extend what a person can do while in use, without necessarily changing the person’s underlying capability. The trial tests that proposition in a specific consulting setting and with particular tasks; it is not a test of every form of workplace learning or every AI system.
“The researchers found “no evidence” that the ChatGPT group performed better on the technical questions completed without the tool.”
— The study’s authors
Limits of the Skill-Retention Test
The trial establishes results for one firm, one set of tasks and one short training session. It does not show whether longer instruction, repeated AI use, feedback from human experts or different kinds of work could produce lasting learning. Nor does the follow-up establish that participants retained no knowledge at all: the authors’ reported conclusion is that they found no evidence of a performance advantage over the control group on the questions tested.
The main analysis included 487 of the 986 consultants initially allocated to groups, after fewer participants began the survey or submitted both tasks. The supplied study summary does not explain in detail how attrition may have affected the estimates. It also does not provide a basis for generalizing the results to other employers, occupations, AI products or later versions of ChatGPT.
The authors’ finding about increased confidence in ChatGPT’s ability to solve tasks it could not solve is identified as exploratory. Its size, durability and relevance outside the experiment remain uncertain. The study likewise does not establish what AI-supported work means for long-term hiring, productivity or business outcomes.
What Employers Should Measure Next
The study does not announce a follow-up trial or a specific next research milestone. Further research would be needed to test whether different training designs or repeated practice lead to skills that workers can use without AI. Results from other organizations and types of work would also help show how broadly the BCG findings apply.
For employers, the immediate question is how to evaluate both AI-assisted output and unaided capability. The trial supports measuring performance when tools are available and separately checking whether employees can explain, verify or reproduce important work without them. Until there is evidence on longer-term learning, companies should avoid treating a higher AI-assisted score as proof of durable technical proficiency.
Key Questions
What did the BCG study find about ChatGPT and data-science tasks?
Consultants given ChatGPT and brief training scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. They remained below the average of BCG data scientists across all three tasks.
Did consultants retain the skills after ChatGPT was removed?
In a later assessment completed without ChatGPT, the researchers found no evidence that the ChatGPT group performed better than the control group. That means the study did not detect a group advantage on its follow-up questions; it does not prove that participants learned nothing.
How many consultants took part in the main analysis?
487 consultants submitted work on both of their assigned tasks and were included in the main analysis. The study initially allocated 986 consultants between the treatment and control groups.
Can the findings be applied to all workers or AI tools?
Not from this trial alone. It involved consultants at one firm, particular data-science tasks, GPT-4 and a short training session. Whether other training methods, jobs or AI systems produce different results remains unclear.
Source: ThorstenMeyerAI.com
