The Problem
I had a literature problem that was slowing everything down.
Extracting Paris-law parameters from corrosion-fatigue research papers — C, m, load ratio R, frequency, environment, closure correction status — was running at roughly 45 minutes per paper. Done carefully, with source verification and unit checking, that is the right pace. Done at scale across 30 or 40 papers, it becomes the bottleneck that makes a parametric scatter analysis impractical.
The question was not whether to use an AI extraction tool. The question was more specific and harder: how would I know if the tool was working before I had already committed my research workflow to it?
That ambiguity is what made this decision worth documenting.
Why the Decision Was Genuinely Hard
The risk with AI-assisted parameter extraction is not obvious failure. Obvious failure is easy to catch — a blank cell, an error message, a number that is off by an order of magnitude. The real risk is plausible failure: a value that looks correct, sits in a reasonable range, and passes a casual review, but came from the wrong table in the source paper or was pulled from a specimen geometry that does not match the application.
In a Paris-law regression, a plausibly wrong C or m value does not announce itself. It produces a crack growth curve that is internally consistent, visually reasonable, and wrong in ways that only become visible when a predicted inspection interval fails against physical reality.
A tool that summarizes well but extracts unreliably is worse than no tool at all — because it removes the skepticism that manual extraction forces you to apply, without replacing it with anything trustworthy.
That asymmetry is why the evaluation design mattered as much as the tool itself.
The Evaluation Design
Before running a single paper through Elicit, I built the test structure. The sequence was deliberate:
Step 1 — Choose a ground-truth test set
I selected 10 papers I had already read and manually extracted from. This eliminated the confound of “I did not know this parameter existed” and isolated extraction accuracy and time cost as the only variables being tested. Elicit’s output would be compared directly against a verified table I had built myself.
Step 2 — Define the extraction schema
Nine fields per paper: material designation, environment (pH, NaCl concentration, temperature), load ratio R, frequency, ΔK range tested, Paris-law C, Paris-law m, whether closure correction was applied, and measurement method. These were not chosen arbitrarily — they were the exact fields where run-to-run scatter had been the diagnostic problem throughout the research.
Step 3 — Set the pass criteria before running the test
This was the most important step and the one most easily skipped.
Two explicit criteria, defined in advance:
- Accuracy criterion: Fewer than 10% of numerical cells may contain an incorrect value — wrong number, wrong parameter, or value pulled from the wrong context. This is the reliability criterion.
- Cleanup criterion: Fewer than 25% of cells may require any formatting intervention before the table is analysis-ready. This is the labor criterion.
These two criteria are not interchangeable. A cell that extracts C = 1.4×10⁻¹⁰ but formats it as “1.4 E-10” costs 30 seconds to fix. A cell that extracts m = 3.1 when the paper reports m = 2.1 — because Elicit pulled from the wrong table — corrupts the regression silently if it is not caught. The accuracy bar is stricter because the downstream cost of failure is higher.
Step 4 — Build the decision tree before seeing results
Three verdicts defined in advance, with no room for post-hoc rationalization:
- Keep and scale: Both criteria met and time savings ≥ 2× manual extraction rate. Expand to full corpus.
- Keep with modification: Accuracy acceptable but time savings marginal. One-week retest with adjusted schema before final decision.
- Drop: Fewer than 9 of 10 papers correctly extracted on core numerical fields. Move immediately to the fallback — Claude with a structured JSON extraction prompt enforcing explicit null-handling.
The decision point was fixed at end of Week 1. Not after two weeks of hoping it would improve.
What Claude Helped Me Clarify
When I brought the evaluation framework to Claude for review, the most useful output was not a new methodology. It was a terminology correction that had real consequences.
The original plan used “80% accuracy” as the pass criterion — a single threshold covering both wrong values and formatting issues. Claude identified the conflation:
“A cell might be extracted correctly but need minor reformatting, or it might be wrong. Those are not equivalent failures. A silently wrong C or m value corrupts a regression; a reformatting issue costs seconds. The accuracy bar should be stricter because the downstream cost of failure is higher.”
That distinction — accuracy as a reliability criterion, cleanup as a labor criterion — changed the evaluation structure in a way that mattered. A tool could pass the original 80% threshold while failing on reliability specifically. The revised two-criteria framework made that failure mode visible before it could cause a problem downstream.
The prompt that produced this was constrained deliberately:
“My integration plan says ‘fewer than 20% of extracted cells require manual correction.’ Review this and tell me whether that threshold conflates accuracy and cleanup burden — and whether conflating them would produce a misleading pass/fail verdict for a quantitative research task.”