These are the publicly available benchmarks the market uses to evaluate AI. They are a starting point. The best evaluations are built on custom datasets grounded in specific workflows. That is what Caliper Lab specialises in.
Whether a model can answer specific questions from SEC filings using the source document as context.
Open-book QA on real 10-K, 10-Q, and 8-K filings. Designed as a minimum performance standard for financial LLMs. Built by PatronusAI with annotated evidence strings.
View datasetWhether a model can locate and interpret financial disclosures across the narrative sections of 10-K filings.
QA pairs grounded specifically in 10-K SEC filings with supporting context passages. Covers MD&A, risk factors, and business descriptions.
View datasetWhether a model can combine data from tables and prose in the same financial report to answer multi-step numerical questions.
Hybrid QA over real 10-K and 10-Q filings. Requires numerical reasoning across tables and text. Human expert baseline: 90.8% F1.
View datasetWhether a model can handle scanned or image-based financial documents, the format most real-world trade and legacy documents arrive in.
Document visual QA extension of TAT-QA. Questions over page images of financial reports. Relevant for vendors processing scanned or low-quality documents.
View datasetWhether an AI agent can retrieve specific numeric values from structured financial databases versus searching unstructured sources.
Agent retrieval benchmark built by Daloopa across 14 agent configurations. Key finding: structured API access outperforms web search by 71 percentage points. Full tool call traces released.
View datasetWhether an agent can identify the right document type and locate the key passage, separating document selection from passage retrieval.
Agentic retrieval benchmark for financial QA on S&P 500 firms. Expert-annotated. Explicitly separates document selection from passage retrieval. ACM ICAIF 2025.
View datasetWhether a model correctly classifies sentiment in financial news text, the baseline for any research or monitoring product making tone or signal claims.
Sentiment classification for financial news sentences. Annotated by 16 people with finance backgrounds across three classes. Most widely used financial NLP baseline.
View datasetWhether a model holds context across a conversation. Follow-up questions about the same filing, where each answer depends on the last.
Multi-turn financial QA over earnings reports. Every other financial set here is single-shot. This is the only one that tests whether reasoning survives a follow-up question.
View datasetWhether a model can answer questions from spoken earnings-call transcripts rather than written filings.
QA over earnings call transcripts. A different document type from a 10-K: spoken, hedged and unstructured, which tends to expose models tuned to clean filing prose.
View datasetQuantitative business reasoning drawn from professional exams and filings, closer to what an analyst is actually tested on.
Eight quantitative reasoning tasks spanning professional certification exams, earnings reports and financial programme code. Broader than document QA alone.
View datasetWhether a model can find and reason over the right numbers in a full-length filing, not a pre-selected passage.
FinQA questions re-set against entire documents averaging around 120 pages. Every other financial set here hands the model the relevant excerpt. This one does not, so it separates retrieval from reasoning.
View datasetWhether a model can tag numeric facts in filings with the correct XBRL label, the task behind regulatory financial reporting.
Entity tagging across 139 XBRL categories on real filings. A structured labelling task rather than question answering, which is closer to how reporting pipelines actually consume a filing.
View datasetWhether a model can identify and extract specific clause types from commercial contracts, the core task in M&A diligence and contract review.
Contract Understanding Atticus Dataset. 13,000+ expert annotations across 41 clause types. Built over a year by legal experts. Gold standard for contract clause extraction.
View datasetWhether a model can reason across a wide range of legal task types, from contract analysis to statutory interpretation.
162 legal reasoning tasks from 40 contributors including lawyers, law professors, and practitioners. Stanford / HazyResearch.
View datasetWhether a model can classify whether a contract clause supports, contradicts, or is silent on a specific obligation, the core NDA and compliance review task.
Natural language inference over contracts. Three-way classification across NDA text. Used across multiple legal benchmarks as a foundational test.
View datasetWhether a model retrieves the right passage from a legal contract before answering, testing retrieval quality separately from generation quality.
Retrieval-augmented generation benchmark specifically for legal contracts. Built by ZeroEntropy to isolate whether the retrieval layer finds the right clause.
View datasetWhether a model correctly identifies legal risk in contract clauses, including detecting laziness where models avoid flagging risk rather than analysing it.
Clause-level legal risk identification built on CUAD. Measures correctness (F1), output effectiveness, and laziness rates across 19 models. April 2025.
View datasetWhether a model can answer questions about consumer terms and conditions rather than negotiated commercial contracts.
Yes/no questions over consumer-facing agreements. A different buyer and a different drafting style from the M&A and NDA material in CUAD and ContractNLI.
View datasetWhether a model can read merger agreements, and find the specific deal terms an M&A lawyer is paid to identify.
Merger Agreement Understanding Dataset, annotated by the Atticus Project. The most directly relevant legal set for a private-equity or corporate-finance buyer.
View datasetWhether a model can summarise legislation: statute and regulation rather than private contracts.
US Congressional and California state bills with human-written summaries. Extends legal coverage from contracts into regulatory text, where compliance work actually sits.
View datasetThese are the benchmarks frontier models are evaluated against. Useful as a general capability baseline. Do not mistake high scores here for task-specific performance on professional workflows.
Broad academic knowledge across 57 subjects including finance, law, and economics. The standard measure of frontier model breadth.
Measuring Massive Multitask Language Understanding. Most frontier models now score above 85%. Useful as a general baseline, not a workflow performance measure.
View datasetMulti-dimensional model evaluation across accuracy, calibration, robustness, fairness, and efficiency. The methodology reference for rigorous evaluation design.
Holistic Evaluation of Language Models from Stanford. The gold standard for multi-dimension evaluation methodology.
View datasetTask types designed to exceed model capability at release. Useful for identifying categories where frontier models still fail.
Beyond the Imitation Game benchmark. Useful for identifying the failure frontier rather than measuring average performance.
View datasetWhether a model can analyse spreadsheets, financial statements, and business reports across QA, data visualisation, and file generation tasks.
AI Data Analytics Benchmark evaluating LLM agents on document analysis spanning spreadsheets, financial statements, and business reports across three task dimensions.
View datasetA single reference point for the full financial AI evaluation landscape across information extraction, QA, forecasting, risk management, and decision-making.
HuggingFace-hosted leaderboard covering 42 financial datasets across 7 categories. Updated continuously by the FINOS Foundation.
View datasetA harder, cleaner successor to MMLU with ten answer options instead of four and much of the original label noise removed.
Frontier models have saturated MMLU, so it no longer separates them. MMLU-Pro still does. Our tracker already collects published MMLU-Pro scores, so results here can be read against the numbers vendors report.
View datasetMulti-step reasoning over passages of around a thousand words. Long-context reasoning rather than recall.
Algorithmically generated murder mysteries, object placement and team allocation problems. Long enough that a model has to hold and combine facts, not retrieve one.
View datasetReading comprehension that requires arithmetic over the passage: counting, sorting and subtraction, not extraction.
The closest general-domain analogue to the numerical work our financial sets measure, which makes it a useful control: it separates document reasoning from domain knowledge.
View datasetConsulting, advisory, and complex multi-document workflows have no meaningful public benchmarks. The best evaluations are built on custom datasets grounded in real tasks. Caliper Lab designs and builds evaluation datasets for AI teams that need to measure what actually matters in their domain.