🔍 Read the full analysis: OpenAI Agent Training Inside Your Software: The Ironclad Fine Print Explained on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI says it trained GPT-6 Astra using practice tasks in hosted copies of Ironclad’s contract-management software. Astra met an average of 55% of task criteria, while estimated completion times were simulated; the results do not establish customer productivity gains or readiness for unsupervised contract work.
OpenAI said on October 6 that it trained its frontier model GPT-6 Astra on legal, commercial and procurement tasks inside hosted copies of Ironclad’s contract-management software. The results show an average of 55% of evaluation criteria met, according to OpenAI, not 55% of tasks completed; the company also says the reported time estimates are simulated rather than measured customer savings.
OpenAI and Ironclad selected 11 tasks with employees who use the software and people familiar with the work. Examples included setting up nondisclosure agreements, creating procurement approval processes and updating a reusable contract clause to reflect a requester’s jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes on each task.
Tasks were scored against eight to 50 criteria, depending on complexity. OpenAI reported that GPT-6 Astra met an average of 55.0% of those criteria, compared with 41.6% for GPT-5.6 Sol, a model labeled “high.” An internal model used during Astra’s development reached 63.7%. On one highlighted task, Astra met about 94% of the criteria. These figures describe performance on the study’s evaluation tasks, not general accuracy across Ironclad customers’ work.
OpenAI said Ironclad supplied hosted product copies for model practice. It built synthetic training tasks from publicly filed contracts in the U.S. Securities and Exchange Commission’s EDGAR database, which it says were filtered to remove personal information. OpenAI also said the work used no OpenAI customer data, internal OpenAI contracts or non-public Ironclad customer data.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Contract Workflow Accuracy Matters
The results matter because contract and procurement workflows encode business controls, not just routine data entry. A procurement process may need Finance approval above a spending threshold, Security review for certain requests and Legal review when terms are nonstandard. Missing one requirement can send a purchase through without a required check; a partial rubric score does not translate neatly into a partially useful workflow.
OpenAI’s report acknowledges that an agent can lose track of a business rule during a multi-step task and says human oversight remains important. The reported 55% average therefore should not be read as evidence that Astra can independently manage contracts. It indicates progress on a difficult, bounded test, while leaving unanswered whether the system can reliably satisfy every necessary control in real deployments.
For software vendors, the initiative also creates a potential trade-off. Working with OpenAI could help improve agents’ performance in a vendor’s product, but more capable agents may eventually become the way customers interact with that product. In that scenario, a vendor’s lasting value may depend less on its screens and more on its business rules, records, audit trail and controls—the parts an agent must use correctly.
As an affiliate, we earn on qualifying purchases.
How Ironclad Became a Training Site
The announcement is about Ironclad, the contract-management company, not a new AI framework called Ironclad. OpenAI described the project as an effort to train models to understand business rules, carry out multi-step work in specialized software and check finished work against the original requirements.
OpenAI’s post also invites a small number of software companies to work on tasks current agents cannot reliably complete. It asks prospective partners to bring a specific failure example, subject-matter experts, a secure testing environment and data that can safely be used for research. The arrangement would place model training closer to the workflows and constraints found inside professional software.
The reported completion times need separate treatment from the rubric scores. OpenAI gave estimates of 37.0 minutes for GPT-5.6 Sol and 19.2 minutes for GPT-6 Astra, but said these were simulated using assumed processing and generation speeds. They were not observed time savings for customers, and the task set was limited to the 11 research exercises.
As an affiliate, we earn on qualifying purchases.
What the Evaluation Does Not Show
The published results do not establish how Astra would perform on live customer contracts, on workflows beyond the 11 selected tasks or when requirements conflict or change mid-task. OpenAI’s report gives average criterion scores, but the source material does not provide a full breakdown of which criteria were missed across tasks. It is also unclear how performance varies by task type or how often a missed requirement would create a material business risk.
The time estimates are not measured productivity results, and the reported scores do not show that an agent can replace expert review. OpenAI says human oversight remains important. No customer deployment outcomes, independent evaluation or schedule for wider commercial availability are provided in the supplied material.
AI-powered contract analysis software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What OpenAI’s Software Partners Need
OpenAI says it is seeking a small group of software-company partners for further work. The next step described is for partners to provide a specific task agents fail at, experts who understand the workflow, a secure environment for testing and research data that can be used safely.
For companies considering agent access to contract, finance or customer systems, the practical follow-up is to ask vendors which requirements the model missed, how permissions and approvals are enforced, and whether each action is logged for review. The supplied announcement does not set a date for the next results or specify when systems trained through this program might reach customers.
enterprise contract automation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did OpenAI announce about Ironclad?
OpenAI described training GPT-6 Astra on 11 contract, legal, commercial and procurement tasks using hosted copies of Ironclad’s software. It also invited a small number of software companies to propose similar research partnerships.Does a 55% score mean Astra completed 55% of the tasks?
No. OpenAI reported that Astra met an average of 55% of evaluation criteria across the tasks. That is not the percentage of tasks completed, and the average does not show which individual requirements were missed.Did Astra cut contract work time in half for customers?
The reported 19.2-minute figure is a simulated estimate, not a measured customer result. OpenAI compared it with a 37-minute simulated estimate for GPT-5.6 Sol and said the figures rely on assumed processing and generation speeds.What data did OpenAI say it used?
OpenAI said it created synthetic training tasks from publicly filed contracts in the SEC’s EDGAR database and filtered them to remove personal information. It said the work did not use OpenAI customer data, internal OpenAI contracts or non-public Ironclad customer data.Can companies use the agent for contracts without human review?
The reported evaluation does not establish that. OpenAI’s post says human oversight still matters, and the average criteria score leaves reliability on specific live workflows unproven.Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
