# Introducing BenchBench: Testing the parsing benchmarks

> BenchBench examines how small changes to document pages affect benchmark rankings.

Canonical: https://reducto.ai/blog/benchbench-document-parsing-benchmarks

Benchmarks are meant to make systems easier to compare. For document parsing, that means testing a parser’s ability to preserve a document’s text, values, tables, and relationships between elements.

A leaderboard further condenses that benchmark into a score and a ranking. Yet it rarely tells you how fickle that order can be, or the factors that led to the composite score. If you are using the result to choose a parser at face value, you’re leaving valuable information on the table.

We ran eight parsers across 12 public benchmarks, using original document pages and the same pages tilted by 2 to 4 degrees. We scored the outputs with each benchmark’s own scorer and a judge that reads the page image. We call the study BenchBench, because we think benchmarks deserve testing, too.

<div className="bp-framed">

![A slightly crooked page reshuffles the leaderboard: 7 of 8 parsers change rank after a 2–4° tilt. Each benchmark’s top score = 100%, separately before and after tilt. Mean across 12 benchmarks.](/blog/benchbench/leaderboard-reshuffle.png)

</div>

<h2 data-toc="How we tested">Our Methodology</h2>

We sampled up to 500 pages per benchmark using a fixed seed. We ran Reducto, LlamaParse, Datalab, Extend, Mistral OCR, Azure Document Intelligence, Pulse, and Cohere, with one configuration per parser.

We then repeated the runs on tilted pages (using ParseBench’s own tilt protocol), simulating a common real-life scenario: documents arrive imperfect.

We then graded each output two ways:

-   The benchmark’s own scorer, using its native metric and rules.
-   [PAIR](/blog/pairwise-ai-ranking-parse-evaluation), an LLM judge, which sees the original, upright page image and all eight outputs with vendor names hidden. It lists errors before assigning scores from 1 to 10. More details about PAIR are in our FAQ, including caveats about our involvement in its development.

<h2 data-toc="What a slight tilt changes">Our findings</h2>

### Small changes reorder the leaderboards

Across all tested parsers and benchmarks, the median relative score drop after tilting was 2.3%. Yet first place changed on 8 of 12 benchmarks, showing just how narrow the margins are at the top.

<div className="bp-framed">

![No parser’s ranking is immune to a 2–4° page tilt. Places each parser gained or lost, one dot per benchmark. 12 benchmarks per parser; each benchmark uses its own scorer.](/blog/benchbench/rank-changes.png)

</div>

On 9 of 12 benchmarks, the original gap between first and second was smaller than the median parser’s score loss after tilting. This shows the minuscule differences that benchmarks are working with: small variations in the document sets, even with the exact same content, can drastically change the results.

<div className="bp-framed">

![On 9 of 12 benchmarks, the tilt penalty exceeds the winning margin. Relative native score changes after a 2–4° tilt. Margin relative to original winner; tilt loss relative to each parser’s original score. Median across eight parsers.](/blog/benchbench/tilt-penalty-vs-margin.png)

</div>

### Preserving data can still result in a zero

In a ParseBench-2 regression table, a parse of a table preserved every coefficient and standard error but placed each standard error in its coefficient’s cell rather than on a separate row. It scored zero. Another parse preserved all text and strikethroughs on a strikeout page but omitted decorative styling. It also scored zero. You can inspect [both examples](https://parsejudge.com/parsebench-2).

In both cases, the content was preserved in its entirety; a human annotator would have done no better. However, strict grading rules mean that it can appear as though these parsers missed the mark entirely.

These cases show why both benchmarks and their individual scores deserve full context: a representation mismatch can receive the same score as missing information.

<h2 data-toc="Why the grader matters">Why we instead grade with a judge</h2>

The goal of parsing is to preserve the page’s information: the right values, in the right cells, in the right order. A judge that reads the page can evaluate that directly, including when outputs represent the same information differently.

We checked [PAIR, our open parse evaluation framework](/blog/pairwise-ai-ranking-parse-evaluation), against people in a separate human study. An outside annotation team reviewed 300 pairwise comparisons, with at least three ratings per comparison and a reviewed human verdict. When both PAIR and the human verdict picked a winner, they agreed 84% of the time.

Under PAIR, the tilt changed first place on 1 of 12 benchmarks. It still penalized degradation: its median relative score drop was 4.4%. Lower ranks moved under both grading methods.

We also tested two short prompts without our detailed rubric. They picked the same winner as our main judge in 25 and 26 (out of 26) benchmark orientation cases. Alternate models (Claude Fable 5.2, GPT-6 Astra, Gemini 4 Argon) varied more, agreeing in 25, 24, and 18 cases.

Judges also have choices worth inspecting: our rubric penalizes missing figure descriptions, and model choice can affect the outcome.

Rule-based scorers also have their merits, especially when exact requirements need to be met. Ultimately, our goal is not to say one methodology is the best, but to be critical of off-the-shelf evaluations and document sets made by vendors, including our own.

<h2 data-toc="Run Your Own Parse Evaluations" data-toc-href="/blog/pairwise-ai-ranking-parse-evaluation#run-your-own-parse-evals">Conclusion</h2>

For anyone choosing a parser, the practical lesson is to look beyond the graphic, and instead to check the differences in results, inspect the pages behind low scores, and most importantly, test on your own documents. That will require more work, but will yield better results every time.

We recommend [ParseJudge](https://parsejudge.com/) to compare outputs, with vendor names hidden, and read the errors behind each score. Those errors will tell you more about fit than a leaderboard position alone, especially for a use-case you know best.

<h2 data-toc="Questions and methodology">FAQ</h2>

### Why a tilt? Real documents are not tilted on purpose.

A 2 to 4 degree tilt is common in phone photos, copier scans and faxes. It is also mild: the text is still easy to read. It is not our invention: we used the tilt protocol from ParseBench. If a small, common change reorders a leaderboard, the leaderboard is sensitive to things that do not matter much to a reader.

### You wrote the judge and sell a parser. Why trust the results?

We sell Reducto, and our parser wins most comparisons under the judge we wrote. You should scrutinize those results. We recommend testing on your own documents: run [ParseJudge](https://parsejudge.com/) on a sample of your pages and read the errors behind each score. The judge checks the outputs without vendor names, so it evaluates each parse without knowing who produced it.

### Can I reproduce this?

Yes. We release everything:

-   The fixed random seed we used to sample every page, and the page lists it gives.
-   The tilted pages.
-   Every parser output.
-   The judge prompts and every verdict, including the alternate judges.
-   The exact code we ran, from page sampling to scoring.
