Introducing Pairwise AI Ranking (PAIR): an open-source framework for document parsing evaluation

Evaluating and choosing a document parser is surprisingly difficult. At Reducto, we have spent thousands of hours poring over the long tail of document data, and assessing different ways to scalably and reliably measure the quality of parses. Honing our evaluation methodology has led us to develop industry-leading models like r-1 and r-1 mini that perform well across diverse enterprise use cases.

Now, we are publicly releasing our Pairwise AI Ranking (PAIR) framework, so everyone can run trustworthy and generalizable parsing evaluations. Our approach focuses on what we found truly matters in practice for preserving document content faithfully.

Why are parse evaluations hard?

Evaluation methodology for document parsing should be unbiased, scalable, and robust across datasets. Most importantly, it must correlate with downstream consequences of parse quality.

Existing parse benchmarks rely on deterministic rules or human-labeled ground truth to assess parses. These rules typically codify small subjective judgement calls about how document information should be represented, many of which do not actually impact a human or agent’s understanding of a document’s content.

For example, a table can be represented equally well in Markdown or HTML with very little downstream effect, but placing a value under the wrong column header changes its meaning. Strict rules are too brittle to cover edge cases found in real-world documents.

Policies written (000s)

| Personal | Q1 | Q2 |
|---|---|---|
| Auto | 42 | 45 |
| Home | 18 | 21 |

| Business | Q1 | Q2 |
|---|---|---|
| Fleet | 7 | 8 |
| Cyber | 3 | 5 |

Policies written (000s)

|  | Q1 | Q2 |
|---|---|---|
| **Personal** | | |
| Auto | 42 | 45 |
| Home | 18 | 21 |
| **Business** | | |
| Fleet | 7 | 8 |
| Cyber | 3 | 5 |

| Policies written (000s) | | |
|---|---|---|
| **Personal** | Q1 | Q2 |
| Auto | 42 | 45 |
| Home | 18 | 21 |
| **Business** | Q1 | Q2 |
| Fleet | 7 | 8 |
| Cyber | 3 | 5 |

The purpose of parsing is to make document information usable by preserving both its content and the relationships that give it meaning. In workflows such as claims processing, financial analysis, and healthcare management, even a single decimal point error can directly affect downstream decisions people and agents make.

Parse quality is best assessed by whether or not information is faithfully preserved from a document. That’s why we developed Pairwise AI Ranking (PAIR), an LLM-as-judge methodology focused on assessing content faithfulness and preservation.

Pairwise AI Ranking (PAIR)

PAIR uses an LLM judge to compare two parses of the same page. The judge sees the original page and both parses, then chooses which parse better preserves the document’s information, or declares a tie. We select a shared baseline parser, then run pairwise comparisons between each candidate’s output and the baseline’s output on every benchmark page. We repeat each comparison three times and report win rates with ties counting as half a win. To reduce position bias, we vary which parse appears first across comparisons and hide parser identities from the judge.

| Item    | Qty | Amount |
|---------|-----|--------|
| Widgets | 12  | $4,200 |
| Bolts   | 40  | $380   |

<table>
<tr><th>Item<th>Qty<th>Amount
<tr><td>Widgets<td>12<td>$4,200
<tr><td>Bolts<td>40<td>$380
</table>

PAIR enables testing of any type of document across any provider without document-specific rules or labeling.

In order to build trust in our judge’s behavior, we investigated how closely judge decisions align with human assessments through a third-party human annotation campaign. We submitted the same 300 pairwise comparisons, covering outputs from four parse providers, to both the judge and human reviewers. Each comparison received at least three independent human ratings, followed by review and adjudication to establish a final human verdict.

As the results below show, the judge’s decisions align with human verdicts when both choose a winner, with stronger agreement when human reviewers are unanimous. The latter is an especially useful test: when every reviewer identifies the same winner, the judge should have that clear preference too.

We also tested whether the LLM judge could discern harmless formatting changes from genuinely damaged content. We edited r-1 outputs for 100 pages in controlled ways and asked the judge to compare each edited version with the original, 604 comparisons in total. Formatting-only edits, like converting an HTML table to a Markdown table, rejoining line breaks, switching to curly quotes, or removing bold and italic tags, were judged as ties with the original every time. On the other hand, edits that fundamentally changed document information were caught – for example, when a single digit changed, two table cells were swapped, or when two paragraphs were swapped, the judge preferred the original r-1 parse 92% - 100% of the time.

Run your own parse evals

Ultimately, the best parser for your team is one that works on your documents. With PAIR, you can evaluate parsers without writing document-specific rules or spending valuable time on human annotations. There’s no longer any reason to rely on someone else’s benchmark to choose your parser.

We’re open-sourcing the evaluation framework here. We also have published ParseJudge, where you can run your own evals, compare providers, and explore parsing results. Bring your own documents and see what parser best preserves the information your workflows depend on.

You are evaluating multiple text extractions of one PDF page.
The original page image is provided for reference.

The parses are blinded and shuffled. Evaluate each parse independently against the page image.

## Instructions
Evaluate each extracted text against the page image.
Rank each parse from best (perfect extraction) to worst (completely wrong or empty), with 0 as the best.
Allow ties: equally good parses receive the same rank, including all parses receiving 0 when equally good.
Use consecutive ranks for distinct quality levels (0, 1, 2, ...).

Here is what you should take into account:

### Text fidelity:
- Stay faithful to the visible page text, even when it appears misspelled, abbreviated, handwritten, awkward, or like a typo. Do NOT autocorrect, normalize, or silently rewrite page text while judging.
- A parse should not be penalized for preserving a visible typo or unusual spelling exactly as shown on the page. Penalize only when the extracted text differs from what is visibly present.

### Tables:
- Do the values in parsed document's cells match the right row, column label pairs visually in the document?
- Are the rows and columns accurately captured?

### Key-value pairs:
- Are the labels and values correctly paired?
- For checkbox fields, are checkboxes correctly parsed as checked or unchecked?
- Can you determine the values of each field from the parsed result?

### Semantic markers and placeholders:
- Do not penalize a parse for using explicit markers that preserve the document's semantics even if the exact marker text is not visibly printed on the page. These include provider-specific tags for blank/empty values, unchecked or checked controls, missing optional values, signature/stamp presence, illegible text, redactions, not-applicable values, or similar normalized states.
- Treat these markers as neutral formatting choices when they make the visible document state easier to recover and do not change the field, table cell, or control's meaning.
- Penalize semantic markers only when they replace visible text, attach to the wrong label or cell, create a false value/state, hide that a required value is present, or make the form/table/control structure materially harder to recover.

### Reading order and block ordering:
- For multi-column pages, text should be cleanly separated into columns, not merged into single lines. This is a catastrophic error.
- The text should read in correct reading order, like how it would be visually inferred by a human.
- Extra, repeated, or visually cluttered Markdown bullet/list markers are formatting noise when they do not change the content, reading order, or parent-child list relationships. Do NOT penalize a parse just because it has duplicated bullets such as "- -", "- o", or bullets before headings if the meaning and hierarchy remain clear.
- Penalize list-marker artifacts only when they create false checkbox states, obscure text, reorder content, flatten meaningful nesting, or make parent/sub-bullet relationships materially harder to recover.

### Figures and images:
- If the extraction includes a text description or summary of a figure, chart, or image on the page, do NOT penalize this. These are intentional figure summaries, not hallucinations or extraction errors.
- Only penalize if the description is factually wrong about what the image shows.

### Hyperlinks and embedded URLs:
- PDFs often carry hyperlink targets (e.g., mailto: addresses, https:// URLs, redirect/safelink wrappers) in a metadata layer that is not always visually rendered as text on the page.
- An extraction that surfaces these embedded URLs is NOT a hallucination -- the URL genuinely exists in the PDF's link layer.
- Treat hyperlink presence vs absence as a neutral formatting choice. Do NOT penalize OR reward an extraction based on whether it includes embedded URLs, markdown link syntax, or raw redirect strings.
- The visible anchor TEXT itself must still match the page; only the inclusion of the underlying link target is neutral.

Ignore differences in whitespace and formatting. Treat curly and straight apostrophes as equivalent.

Return JSON with exactly one evaluation per label listed here:
{labels}

For each evaluation, include the label, a list of specific errors found, concise reasoning, and the zero-based rank.

## Parses to rank
{parse_sections}