The Problem Isn’t the Model, It’s What It’s Reading
Most teams still treat document parsing as a solved problem, you upload a file, turn it into text, and move on to the interesting part.
However the recent work from LlamaIndex in ParseBench puts some pressure on that layer, not by introducing something entirely new but by tightening what good actually looks like. It’s no longer enough that a document is readable, it has to be reliable enough for a system to act on, which is a different bar entirely.
Parsing isn’t neutral
It’s easy to think of parsing as just a technical step that happens before the real logic kicks in, but in practice it’s already shaping the outcome whether you treat it that way or not.
ParseBench looks at areas most pipelines quietly degrade:
- Tables, not just values but structure
- Formatting signals like indentation and deletions
- Faithfulness, meaning no silent changes or omissions
- Visual grounding, linking content back to layout
None of this is particularly exotic, it’s just the stuff people tend to ignore because it’s harder to measure and doesn’t show up immediately.
The issue is once it’s lost, you don’t get it back, so the model isn’t really reasoning over the document, it’s reasoning over a partial reconstruction of it.
Legal documents aren’t just text
This is where it impacts us quite directly in legal.
A contract isn’t a blob of language, it’s a structured argument with rules embedded in how it’s written, things like hierarchy, numbering, cross-references, defined terms, all of that carries meaning in ways that aren’t obvious if you flatten it.
Flatten that and you’ve already changed the document, so the model never actually sees what was really there in the first place.
You see it in small ways that don’t look like much individually but add up quickly:
- A clause boundary changes slightly and now two obligations look like one
- A defined term isn’t linked properly so references drift
- A table is extracted as lines of text and loses relationships
Each of those looks minor but together they change what the system thinks it’s reading, which stops being a parsing problem and starts becoming an outcome problem.
The real issue is more how it fails
What ParseBench shows us I think, is that failures aren’t clean or obvious, like most failure, they tend to sit in that uncomfortable middle ground.
You don’t get a clear error, you get something that looks mostly right.
One system handles tables well but drops formatting, another keeps layout but introduces small inaccuracies, and none are consistently strong across all dimensions.
So what you end up with is drift that’s hard to spot in isolation:
- Slightly wrong structure
- Slightly incomplete content
- Slightly misaligned references
The model still produces an answer and it usually sounds reasonable enough at a glance, but it’s anchored to something that’s already a bit off, which is where things start to go wrong.
It doesn’t break loudly, which is really the issue.
Applying this in a legal pipeline
This isn’t really about swapping one parser for another, it’s about treating parsing as part of the system design rather than a hidden dependency that no one owns or cares about after deploying.
Start with control, because parsing should be something you can inspect, version, and reason about, so if a result is questioned you can reconstruct what the system actually saw rather than just pointing back to the original PDF.
That sounds obvious, but most setups can’t actually do it in practice.
Then define what "good" actually means in your context, because generic benchmarks help but they don’t map cleanly to your legal work.
You need something closer to:
- Clause boundaries that align with how lawyers read the document
- Cross-references that resolve correctly
- Defined terms that stay consistent throughout
- Tables that preserve relationships, not just values
- Redlines that aren’t discarded as noise
Without that, you’re measuring the wrong thing and getting false confidence from it.
Evaluation needs to move up a level as well.
Instead of asking whether the text extraction is accurate, tie it to tasks you actually care about, things like extracting obligations, identifying termination rights, or building a timeline, then compare the result to something known.
If the task fails, the parsing failed and it doesn’t really matter how clean the intermediate output looked.
Confidence is the other piece that’s usually missing.
Not every parsed document should be treated the same, some are clean, some are messy, most sit somewhere in between, and the system should reflect that rather than pretending everything is equally reliable.
High confidence can flow through automation, lower confidence should slow things down, introduce checks, or stop entirely, which is where parsing starts to influence behaviour rather than just input.
More is better
One practical reality that comes out of ParseBench is that no single approach is consistently strong.
Different document types stress different parts of the system, so a clean structured filing behaves very differently to a heavily redlined agreement or a scanned bundle, and treating them the same is where things start to break down.
Relying on one parser is simple, but it’s very fragile.
In practice you need to end up with a mix, either routing by document type or running multiple approaches and reconciling differences, which sure, adds complexity but reflects the reality of the data.
Where this goes
This starts to push parsing out of the background and into something you actively think about.
It becomes part of the audit surface, not just prompts and outputs but the structured representation of the document itself, so if something is challenged that’s what you need to point to.
It also changes how retrieval should work, because chunking flattened text and embedding it only works if the structure is intact, and if it isn’t you’re retrieving from something that’s already degraded.
The direction of travel is fairly clear, more structure, more traceability, tighter links back to source.
But you've not mentioned Agentic
There’s also a knock-on effect for anything agentic.
A lot of current thinking assumes that once models are good enough they can take on more of the workflow, reviewing contracts, extracting obligations, making decisions, all of that depends on the input holding up.
If parsing is unreliable, the system won’t fail in a way that’s easy to catch, however it will act on a slightly wrong view of the document and carry on, which is a harder problem than model capability.
Most effort in legal AI still goes into prompts, models and interfaces because that’s where the visible progress is.
ParseBench is a reminder that the constraint often sits earlier, and if the system doesn’t have a faithful representation of the document then everything built on top inherits that weakness.
Given most real issues in legal work come from process and handling rather than interpretation, this fits quite neatly.
The risk isn’t just that the model gets the law wrong, it’s that the system never properly understood the document in the first place.