What this dataset measures
Natural language inference on NDA clauses: Entailment, Contradiction, or NotMentioned relative to a hypothesis.
Same composite metric set as LegalBench contracts: three active metrics. Hallucination proxy is uniform and excluded from scoring.
Composite ranking across 9 models
Composite score is the unweighted mean of 3 active metrics: Numeric match, Text similarity (F1), LLM judge accuracy.
Groundedness ~0% and hallucination proxy at 100% are artifacts of short classification answers. Excluded from composite.
How models differ across scoring dimensions
Scores below are the metrics that feed the composite for this dataset.
| Claude Fable 5 | 86.7% |
| Claude Opus 4.7 | 83.3% |
| GPT-5 | 82.7% |
| Claude Opus 4.8 | 82.7% |
| Claude Sonnet 4.6 | 82.0% |
| Groq Llama 4 Scout | 82.0% |
| GPT-5 mini | 80.7% |
| Claude Haiku 4.5 | 80.0% |
| Groq Llama 3.3 70B | 76.0% |
Unweighted mean across test rows. Measures whether extracted numbers match the reference answer.
| Claude Fable 5 | 86.7% |
| Claude Opus 4.7 | 83.3% |
| GPT-5 | 82.7% |
| Claude Opus 4.8 | 82.7% |
| Claude Sonnet 4.6 | 82.0% |
| Groq Llama 4 Scout | 82.0% |
| GPT-5 mini | 80.7% |
| Claude Haiku 4.5 | 80.0% |
| Groq Llama 3.3 70B | 76.0% |
Unweighted mean across test rows. Token-level F1 between model output and reference string.
| Claude Fable 5 | 86.7% |
| Claude Opus 4.7 | 83.3% |
| GPT-5 | 82.7% |
| Claude Opus 4.8 | 82.7% |
| Claude Sonnet 4.6 | 82.0% |
| Groq Llama 4 Scout | 82.0% |
| GPT-5 mini | 80.7% |
| Claude Haiku 4.5 | 80.0% |
| Groq Llama 3.3 70B | 76.0% |
Unweighted mean across test rows. LLM judge scores answer correctness against the reference.
Where models go wrong
Representative examples of where models struggle on this benchmark. Each pattern shows a common error type drawn from the evaluation set.
The model incorrectly classifies a hypothesis that is supported by the contract clause as something other than Entailment. This occurs when the supporting language is indirect or requires inference rather than explicit matching. The model pattern-matches surface language rather than assessing whether the clause logically supports the hypothesis.
Watch for: Entailment failures are most common where the contract uses conditional or qualified language. A clause that says "may" rather than "shall" is a common trigger. If your workflow involves NDA compliance checking, test specifically on conditionally-phrased obligations.
When a hypothesis is not addressed in the contract at all, the correct classification is NotMentioned. Models frequently return Contradiction instead, treating silence as active denial. The contract does not say the hypothesis is false; it simply does not address it. This distinction matters for compliance review where absence of a provision is not the same as prohibition.
Watch for: Not-mentioned vs contradiction errors are consequential in NDA and compliance contexts. A clause classified as Contradiction implies the contract prohibits something; NotMentioned implies it is silent. These have different legal meanings. Validate this class boundary explicitly before deploying on compliance workflows.
The model fails to identify that a contract clause actively contradicts a hypothesis, returning Entailment or NotMentioned instead. Operative language, the words that create rights, obligations, or restrictions, is the source of contradiction. The model reads the clause but does not register the operative effect of terms like "shall not", "expressly prohibits", or "waives all rights to".
Watch for: Contradiction misses on operative language are the highest-risk failure on this benchmark for legal review. A missed contradiction means a reviewer believes a provision is silent or permissive when the contract actually prohibits it. Spot-check all NotMentioned and Entailment outputs on clauses containing negative obligations.
What this means for deployment
Claude Fable 5 leads at 86.7% composite across 150 ContractNLI tasks, with a fleet average of 81.8%. ContractNLI tests three-way classification: whether a contract clause entails, contradicts, or leaves unstated a given hypothesis. This is closer to legal risk review than clause spotting: it requires the model to reason about what a clause means, not just whether it exists. The fleet scores are strong but the failure modes carry real legal significance.
Claude Fable 5 leads on composite (86.7%), numeric match (86.7%), and LLM judge accuracy (86.7%), with context groundedness at 0.0%. As with LegalBench, the zero context groundedness is a metric artefact. Models return a classification label (Entailment, Contradiction, NotMentioned), not a quoted span. The metric measures span overlap, which is not meaningful on single-word classification outputs. Composite and LLM judge accuracy are the relevant signals.
Text similarity at 86.7% reflects near-perfect string overlap on correct classifications, since the answer is always one of three fixed labels. Hallucination proxy at 100% is expected: there is no free-text generation to hallucinate in. Context groundedness at 0.0% is a metric artefact on classification tasks, not a grounding failure. The meaningful performance signal is the distribution of errors across the three classification classes, particularly the Not-mentioned vs contradiction boundary where failures have the most legal consequence.
Interactive charts
Cost & workflow analysis
Claude Fable 5 leads on composite (86.7%); several alternatives trade a few points of accuracy for much lower run cost. Groq Llama 4 Scout sits within 4.7 pts of the leader at roughly 60× lower cost — strong value for high-volume or pilot workloads.