🔍 Read the full analysis: The Law Model’s Result In Our 42-Setup AI Test: 8 Out Of 100 on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A Lexassay test of 42 AI setups across 15 legal questions gave GLM-5.3 FlashX Legal a score of 8 out of 100, with no points on the hidden questions. The test operator says the result reflects one response per setup and question without internet or tools, so it is a limited spot check rather than a definitive ranking of legal AI.
GLM-5.3 FlashX Legal scored 8 out of 100 in a test by Thorsten Meyer AI that compared 42 AI setups on 15 legal questions, according to the test operator. The model, which its maker Z.ai describes as optimized for legal workflows, received zero points on the hidden questions; the result raises questions about relying on legal-specialist chatbots, while the test’s own disclosures and limited design call for caution in interpreting its rankings.
Lexassay tested models on questions covering German, U.S. and EU law, including legal deadlines, recent changes, court decisions and prompts based on false premises. Ten questions were public and five were hidden. The models answered without internet access or tools, and each answer was graded against a prepared answer with weighted criteria. Citations were checked against official databases, the publisher says.
According to Thorsten Meyer AI, GLM-5.3 FlashX Legal received 8/100 overall and 0 on the hidden set. The citation review found 21 citations classified as real, 25 as misattributed and one as invented. On two questions about German law, the model used its stated 64,000-token reasoning budget without producing an answer. Its base model scored 3, the test operator reported.
The published results also put Claude Opus 5.5 at 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The report says the leading model fully met all criteria on eight of the ten public tasks. Mistral Large 4 scored 10; the operator attributed much of its lost credit to citation errors. These scores reflect Lexassay’s grading rules and test set, not a general measure of performance across legal work.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal AI Scores Matter
Legal questions can turn on precise details, and an incorrect answer may carry practical consequences. The test’s deadline example illustrates the risk: two German civil cases with judgments served on the same date can have different appeal deadlines because Corpus Christi is a public holiday in Bavaria but not Berlin. In the example, the Munich deadline moves to May 28, 2027, while the Berlin deadline remains May 27. A one-day error could affect whether an appeal is admitted.
The score also draws attention to a less visible failure mode: a fluent answer can cite a real law or ruling while misstating what it says. Lexassay’s rules deduct points for invented and misattributed citations, and the publisher reports 804 citations in those categories across all published runs, alongside 2,644 it classified as real. That combined tally is the operator’s finding; the article does not establish how often such errors occur in everyday legal AI use.
The practical implication is limited but relevant: a model marketed for legal workflows should not be treated as a substitute for checking primary sources or getting professional advice. The test operator recommends verifying every cited law and decision and not taking a legal deadline from a chatbot.
Products Worth Considering
FLEXISPOT EN1 Electric Standing Desk, 60 x 24 in, Black
FLEXISPOT 36-Inch Height-Adjustable Standing Desk Converter, Black
FLEXISPOT 60 × 24-Inch Electric Standing Desk, White
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Thorsten Meyer AI says Lexassay used 15 questions per setup: ten published and five hidden from public release. The hidden set is intended to reduce the chance that future models can learn the answers directly from the web. Questions span three legal systems and include both substantive issues and deadline calculations. A prompt built on a false premise is designed to test whether a model corrects the premise rather than accepting it.
The publisher calls the exercise a spot check, not a strict benchmark, because it used one answer per model and task. It says small score differences may be noise. Different reasoning-effort settings were tested separately, meaning a single underlying model may appear more than once among the 42 setups. The grading deducts a quarter of a task’s points for each invented or misattributed citation and 5% for an answer that lacks a legal-advice disclaimer or referral to a lawyer.
The operator disclosed that Thorsten Meyer AI runs Lexassay and that several Claude agents graded the answers, while a Claude model ranked first. It says the method page and published answers, gold answers and citation checks are available for scrutiny. Those disclosures give readers relevant information about the test’s administration, but do not remove the need to assess its design and scoring independently.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in the Lexassay report
Limits of the Published Scores
The results do not show how consistently a model would answer the same question across repeated attempts. Lexassay reports one answer per model and task, and acknowledges that small score differences may be noise. The source also does not establish how the test results compare with performance by lawyers or with AI systems that can search legal databases and use external tools.
The reported scores depend on the chosen questions, gold answers, citation rules and penalty system. The source says the underlying answers and checks are published, but this article has not independently regraded them. It is also unclear from the supplied material how representative the 15 questions are of the range and frequency of real legal tasks. The results should not be read as proof that a particular system will fail or succeed in a real case.
What Readers Should Check Next
Thorsten Meyer AI says its method page and public answer materials can be used to inspect how answers were graded; the five hidden questions are not disclosed, with only their scores published. Readers seeking to evaluate the findings can review the published questions, gold answers and citation checks, then compare those materials with the stated scoring rules.
No follow-up test date or response from Z.ai is specified in the supplied source. Until further information is published, the operator’s recommendation is to use AI answers only as a starting point, verify citations against the underlying legal sources and consult a lawyer about deadlines, claims or contact with an authority. This test is not legal advice, and its scores do not determine the correct answer in an individual case.
Key Questions
What score did GLM-5.3 FlashX Legal receive?
Thorsten Meyer AI reports that the model scored 8 out of 100 on Lexassay and received zero points on its five hidden questions.
What did the Lexassay test measure?
It compared 42 AI setups on 15 questions about German, U.S. and EU law. The models answered without internet access or tools, and their answers were graded against prepared criteria with citations checked against official databases, according to the test operator.
Does the 8/100 score prove the model is unsafe for all legal work?
No. The publisher describes Lexassay as a spot check, not a strict benchmark, with one answer per model and task. The result reports performance on this specific test and does not establish how the model performs on every legal task or with external tools.
Why does the test emphasize citations and deadlines?
A citation may be nonexistent or may not support the claim made, and a deadline can depend on jurisdiction-specific details such as local public holidays. The report uses these issues to illustrate why legal answers need source checks; it advises readers not to rely on a chatbot for a legal deadline.
Who ran and graded the test?
Thorsten Meyer AI runs Lexassay. The publisher says several Claude agents graded answers blind and discloses that a Claude model ranked first, describing that as a conflict of interest.
Source: ThorstenMeyerAI.com
