🔍 Read the full analysis: 71 AI Configurations Tested On My News Pipeline: My Takeaways on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer says he tested 71 AI model configurations on a news-writing pipeline from Sept. 8 to Oct. 13, 2026, using English and German articles based on real sources. GPT-6 Astra had the strongest reported scores, while he chose GPT-6 Luna for production because it cost far less; he also removed two fallback models after tests found fabrication and invalid output.
Thorsten Meyer says he tested 71 AI configurations on a news-writing pipeline using real sources from Sept. 8 to Oct. 13, 2026, and selected GPT-6 Luna for production despite GPT-6 Astra receiving higher quality scores. Meyer reported that Astra averaged 4.00 out of 5 in the initial English and German tests, while Luna cost about $0.0012 per article compared with roughly $0.12 for Astra; he also removed two backup models after his test found fabricated quotations or unusable output.
The test used Meyer’s production prompt and two source texts: a 6,340-character product filing and a 619-character political news item. Each configuration generated an English and German article. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, scored the work on source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with an additional language-quality measure for German.
Meyer reported that GPT-6 Astra scored 4.00 in both languages in the standard test and achieved 3.93 across a 12-article robustness run. Its source-fidelity score in the standard test was 5.00. GPT-6 Luna scored 3.67 in the robustness run, with source fidelity of 4.71; Meyer said all of Luna’s articles produced valid output under the pipeline’s repair process.
The cost and speed figures shaped the production choice. Meyer put Astra at about $0.12 and 50 seconds per article, compared with Luna at around $0.0012 and 17 seconds. He also reported that raising Luna’s reasoning effort did not produce a reliable quality improvement: the maximum setting cost about 12 times as much and took about 12 times as long as the no-effort setting, while the observed score differences fell within the test’s uncertainty.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Why Pipeline-Specific Testing Matters
The results point to a practical distinction between a model’s general reputation and its performance on a particular newsroom task. Meyer’s findings suggest that leaderboard position alone did not predict article quality in his workflow: Claude Opus 5.5, which he describes as the strongest coding model on a separate benchmark, scored below Luna in the article-writing robustness run. These results apply to this test and do not establish how the models would perform across other prompts, sources or publishing standards.
Cost also matters when articles are produced at volume. Meyer’s reported Astra price was about 100 times Luna’s, while the score gap in the robustness run was 0.26 points on a five-point scale. That trade-off may help publishers decide whether a higher-scoring model justifies its added expense, but the choice depends on the quality threshold and risk tolerance of each operation. The test offers one case study, not a universal ranking.
The reported fallback failures highlight a separate operational risk. A system can have a capable primary model yet still publish poor or fabricated material if it automatically switches to a weaker backup. Meyer said the two removed fallbacks either produced extremely low source-fidelity scores or failed the output-validity requirement. Testing the full model route, rather than only its first-choice model, can reveal failures that an aggregate leaderboard may not show.
As an affiliate, we earn on qualifying purchases.
How the 71-Configuration Test Worked
Meyer described the exercise as an evaluation of models across API, subscription, free-tier and locally run open-weight options, using the same production task: turn a supplied news source into a finished article. The two sources differed substantially in length, allowing the test to include a short input that Meyer said was representative of much of his news workflow.
The two judges saw the original source while scoring each article. Meyer also ran the strongest configurations on 10 additional real sources, mostly short ones, and reported 95% bootstrap intervals for the robustness results. He said that, with only two articles in the standard test, score gaps below about 0.3 points should be treated as noise, making the larger follow-up run more useful for ranking.
An article counted only if it generated output the pipeline could use after the same repair step applied in production. Meyer said he made the decision to remove two backup models on Oct. 7. His account was published on ThorstenMeyerAI.com on Oct. 13, 2026.
“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”
— Thorsten Meyer
As an affiliate, we earn on qualifying purchases.
Limits of the Reported Results
The results come from one publisher’s prompt, pipeline and source sample. Meyer’s standard test used two source texts, and although the robustness run added 10 more articles, the account does not provide enough detail to establish how representative the samples are of other newsrooms, beats, source languages or breaking-news conditions. The reported scores therefore should not be read as a general performance ranking for all news-writing uses.
The assessment also relied on two AI judges, one of which was Claude Opus 5.5, a model included in the comparison. Meyer reports their scoring method and uncertainty intervals, but the account does not describe an independent human-editor evaluation or provide the underlying articles and full scoring records here. It is not clear how model versions, prices or results may change after the test period, or whether the findings have been independently replicated.
cost-effective AI content creation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Production Route After Testing
Meyer says GPT-6 Luna is now the production choice and that the two weak fallback models, GPT-4.1 nano and gpt-oss-120b, were removed from the news route on Oct. 7. The source account does not specify a replacement fallback or describe the complete current failover chain, so its final configuration remains unclear.
The next useful evidence would be performance across a larger, continuing set of real stories, including routine checks for fabricated details, invalid output and changes in cost or latency. Meyer’s report does not announce a future test date. For now, the stated decision rests on the results he gathered through Oct. 13, 2026, rather than on a long-term comparison.
AI language model for news writing
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
Which model scored highest in Meyer’s test?
Meyer reported that GPT-6 Astra scored 4.00 out of 5 in both the initial English and German tests. It averaged 3.93 across the 12-article robustness run.
Why did Meyer choose GPT-6 Luna for production?
Meyer said Luna delivered a lower reported quality score than Astra but cost about $0.0012 per article, versus about $0.12 for Astra. He described Luna’s results as consistent and said every Luna article in the test produced valid output.
Did higher reasoning effort improve the articles?
In Meyer’s reported tests, score differences across Luna’s six reasoning-effort settings were within the noise he identified. The maximum setting took and cost about 12 times as much as the no-effort setting, without a reliable quality gain.
What happened to the fallback models?
Meyer said he removed GPT-4.1 nano and gpt-oss-120b from the news route on Oct. 7. He reported that judges flagged invented quotations in GPT-4.1 nano’s output, while gpt-oss-120b produced no valid articles in four attempts.
Do these results show which model is best for every newsroom?
No. The findings reflect one pipeline, one production prompt, a limited set of sources and two AI judges. Other publishers would need to test models against their own reporting standards, sources, languages and costs.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
