Autoregressive models can get stuck in repetition loops. A phrase appears once, then again, and each repetition makes the next one more likely. Since every generated token becomes context for the next prediction, the loop can reinforce itself until the model emits an end token or hits its generation limit. Here’s a useful explanation of why this happens. This is an issue for two reasons: first, the output itself is bad and unusable because of the repetitions and requires a fix, and second, it negatively impacts latency, since we waste time generating excessive tokens that won’t be used anyway.
Below is an example of a standard repetition loop from our document parsing model:
<tr>
<td>#REF!</td>
<td>#REF!</td>
<td>#REF!</td>
</tr>
<tr>
<td>#REF!</td>
<td>#REF!</td>
<td>#REF!</td>
</tr>
<tr>
<td>#REF!</td>
<td>#REF!</td>
<td>#REF!</td>
</tr>
<tr>
<td>#REF!</td>
<td>#REF!</td>
<td>#REF!</td>
</tr>
<tr>
<td>#REF!</td>
<td>#REF!</td>
<td>#REF!</td>
</tr>
...The source page contains that row twice. The model keeps emitting the exact same row indefinitely.
<tr>
<td>S10</td>
<td>100'</td>
<td>1</td>
<td>1000W</td>
<td>1000W</td>
</tr>
<tr>
<td>S11</td>
<td>100'</td>
<td>1</td>
<td>1000W</td>
<td>1000W</td>
</tr>
<tr>
<td>S12</td>
<td>100'</td>
<td>1</td>
<td>1000W</td>
<td>1000W</td>
</tr>
<tr>
<td>S13</td>
<td>100'</td>
<td>1</td>
<td>1000W</td>
<td>1000W</td>
</tr>
<tr>
<td>S14</td>
<td>100'</td>
<td>1</td>
<td>1000W</td>
<td>1000W</td>
</tr>
...Another example of repetition, where it is not an exact repetition, but still a degenerate output.
This is a catastrophic failure even when the output syntax is valid. The parser returns content that was never in the document, generates thousands of unnecessary tokens, and holds an inference slot until it reaches the token limit. If the system notices only once we hit the token limit, it has already paid almost the full latency and compute cost of the failure.
Why document parsing is especially exposed
Fast document parsing needs models that understand complex layouts while remaining economical to serve at high volume. Small VLMs are an attractive option here, but degenerate repetition is an unavoidable tail risk. Better models and training reduce its frequency, but it is still not safe to ignore.
Document parsing also differs from open-ended generation. Standard generation can actually benefit from a higher temperature. This increases the randomness in generation and can help the model be creative and generate diverse outputs. The higher temperature also reduces the likelihood of falling into a repetition loop. For document parsing, given a page there is usually one preferred, deterministic transcription and structure. Sampling with a higher temperature for this use case may also help escape a loop, but it can make the output less accurate.
The olmOCR 2 paper describes the same tension: lower temperatures improved OCR quality, but increased the risk of repetition loops. We also observed this behavior on our internal benchmark, and got our best accuracy at a very low temperature. Rather than increase temperature and compromise every generation to protect against a rare failure, we wanted to identify the bad generations and handle only those differently.
This problem has two main requirements:
- Detect repetition early, before it consumes the full token budget.
- Maintain high precision. Real documents, especially forms and tables, may contain genuine repetition, and we want to minimize false positive detections on such cases.
The second requirement is what makes this harder than searching for duplicate strings. Some forms contain the same blank section several times, or some tables contain many identical rows. A detector that stops whenever it sees repeated HTML will destroy valid parses.
Starting with vLLM’s repetition detector
vLLM includes an exact token-pattern detector controlled by three main parameters:
min_pattern_size: the shortest repeated sequence to consider. Lower values catch tiny loops, but create more chances to flag repeated punctuation or empty cells.max_pattern_size: the longest repeated unit to consider. Higher values catch rows and paragraphs, but make legitimate repeated templates harder to distinguish from degeneration.min_count: how many consecutive copies are required. Lower values stop failures earlier and improve recall; higher values protect precision.
The approach we test here is simple: find an operating point which has good precision and recall on our distribution of generations. To test this, we first parsed 40,000 pages with our parsing VLM. We disabled repetition stopping and allowed generations to run up to 32,768 tokens. In that run, 204 pages hit the full cap, a strong sign that the generation had become stuck.
We then replayed a large grid of detector configurations over the saved output tokens, and observed which generations were flagged at different repetition detector parameter values.
On our internal document distribution, the best conservative vLLM policy produced 100% precision and 98.0% recall. It found the repeating generations at a median of token 1,813. Compared with waiting for the 32,768-token cap, that saved a median of 30,955 generated tokens per detected failure. Even against an 8,192-token cap, it would save about 6,400 tokens.
That is a strong result, but the 98% recall is not as good as it might first appear. A 1-2 percent failure mode becomes thousands of bad pages when a system processes millions of pages. “We catch almost every catastrophic output” is not a satisfying production guarantee.
What the detector missed
Looking at the misses revealed two distinct failure modes.
The first was long-period repetition. One output repeated a 936-token table section four times. A detector capped at short patterns cannot see that loop. Simply increasing the maximum pattern size is not enough: if every pattern must repeat dozens of times, a long block will reach the model’s token limit before it triggers.
The second failure mode was repetition with predictable changes:
2074-2075-2076-2077-2078...or:
T70 -> T147[Thought] -> T148[Answer]
T72 -> T149[Thought] -> T150[Answer]
T74 -> T151[Thought] -> T152[Answer]
...These outputs are clearly degenerate to a human, but their tokens never repeat exactly. Numbers, coordinates, letters, or identifiers change on every cycle. We saw arithmetic progressions, Excel-style counters such as A ... Z -> AA, and nested counters such as 8.1A ... 8.1Z -> 8.2A.
The precision problem appears here too: real documents can contain long, orderly sequences of numbers.
We added adaptive min_count detection for long repeated blocks and a custom detector for predictably changing patterns. The serving integration took additional engineering because detection has to happen during generation, not as a post-processing pass. Inference engines don’t allow custom repetition detection logic out of the box. vLLM’s custom logits processor can be used, but currently does not work alongside MTP.
We later expanded the study to a repetition-heavy 25,000-page corpus to put more pressure on the edge cases. On the reviewed evaluation set for the combined detector, we observed:
- 1,589 true positives, 100% recall on the observed set
- 2 false positives, 99.87% precision
The custom paths caught 34 pathological outputs that the native detector missed, spanning long exact patterns, numeric progressions, and composite counters.
“100% recall” means no misses in that reviewed set, not that a model can never have a new form of repetition. We continue collecting and reviewing production samples, so we can observe failure patterns as they happen and update our system.
Detection changes the fallback tradeoff
One way to handle repetition is to set a low maximum token limit and fall back whenever the model reaches it. That is safe, but blunt. A legitimately dense page may also need more than 8,192 tokens, so a low cap could send good generations through a more expensive pipeline.
In our 40,000-page experiment, 219 outputs crossed 8,192 tokens. Of those, 204 were pathological repetition and 15 were legitimate long outputs. Treating every 8,192-token generation as a failure would create 15 unnecessary fallbacks: 6.8% of the pages crossing that threshold.
A reliable detector lets us keep a larger generation budget. Repetition is usually stopped around 1,500-1,800 tokens, while dense but valid pages can continue. That saves primary-model compute and avoids adding fallback latency to pages that did not need fallback at all.
Detection does not answer what the fallback should be. Options include sending the page through a different pipeline that is highly unlikely to fail, or retrying the VLM with a different sampling strategy. olmOCR 2’s dynamic temperature scaling is one example of the latter.
Deploying VLMs reliably in production
We keep improving our models and training methods so repetition happens less often in the first place. But at production scale, reducing a failure rate is not enough. Every remaining case still needs to terminate quickly and recover correctly.
Strong models and a strong reliability layer work together. The model handles the overwhelming majority of pages quickly and accurately; the surrounding system recognizes when generation has gone off the rails, and chooses a fallback without making the customer wait. That combination is one of the reasons we are able to serve VLMs reliably at Reducto’s large volume.
Written by Aniruddha Deshpande and the Reducto ML team. Edited by Palak Agarwal.
Additionally, if these types of problems excite you, apply to join our team on our careers page.