Turning up the Quality Dial: Fit-for-Purpose Abstraction in EHR-Derived RWD

A patient with metabolic dysfunction-associated steatotic liver disease (MASLD) has a FibroScan test that shows advanced fibrosis. FibroScans assess liver damage by measuring fibrosis (i.e., liver stiffness measurement, LSM) and steatosis (i.e., controlled attenuation parameter, CAP). The treating hepatologist records the result in a clinical visit note, but no diagnosis code, procedure code, or lab feed records the measurement. To a research dataset built only on structured data, this important information about disease severity doesn’t exist. This matters because LSM often drives cohort selection in MASLD studies. Study criteria may require patients with specific fibrosis stages at index or with serial LSM measurements. But LSM results rarely appear in structured feeds, leading to an unacceptable level of completeness in structured data. To capture it at scale, we need to abstract from unstructured data. Structured data can also mislead in the other direction: a medication order may never be filled, or a diagnosis code entered may be later ruled out.

Gaps like this are common in real-world data, and they matter more as liver research moves away from biopsy and towards non-invasive tests like vibration-controlled transient elastography (VCTE, commonly FibroScan). These gaps are not just in disease staging variables; they also exist across treatment-related variables (e.g., oral treatment, surgical interventions) and outcomes. While filling these gaps is possible, the problem lies in how to fill them with data that is accurate, at scale, and fast enough to support the clinical, regulatory and commercial decisions that depend on it. The solution lies in measuring quality carefully, and spending effort only where it changes the answer.

Quality belongs to a variable

FDA's 2024 guidance on EHR and claims data asks sponsors to assess relevance and reliability not only for the data source as a whole, but for the key data elements in a study, including how exposures, outcomes and covariates were ascertained and validated.

The same variable can be more than adequate in one role and inadequate in another.

When a variable is used to select patients, an error changes who is in the study. A missed FibroScan result means an eligible patient is left out. The study gets smaller, and possibly less representative, but the patients in it are still described correctly. An error in the liver stiffness measure, on the other hand, could result in including a patient in the study who has a liver stiffness value that is higher or lower than the target range.

When a variable is an outcome, errors go straight into the results. Every missed event lowers the observed event rate. If events are missed more often in one treatment group than in another, the comparison itself is skewed.

Other variables can tolerate more error. A confounder is a characteristic, such as age, diabetes or smoking history, that differs between the groups being compared and also affects the outcome. Studies usually balance the groups using a propensity score, a single summary built from many confounders at once. Because each confounder is one input among many, an occasional error has a diluted effect. If a former smoker is recorded as a never-smoker, related factors in the model still carry much of the same information, and that patient's propensity score may not meaningfully moves.

When a study depends on timing, dates matter as much as values. Establishing a patient's fibrosis stage at the start of follow-up, or measuring how liver stiffness changes between tests, requires knowing when each test happened. A correct result with a missing date can be as unusable as a wrong one.

The tension between quality, scale, and speed

Abstracting data from unstructured documents is a well-established approach to enrich electronic health record (EHR)-derived datasets. Abstraction methods differ in how fast and resource intensive they are. At the costly end is dual human abstraction, where two independent human abstractors review each chart and cases of disagreement are resolved by a clinical expert. This approach produces the highest quality data available, but the amount of time spent on each patient limits how large the cohort can be and how quickly the data can be delivered and refreshed.

On the other end of the spectrum is large language model (LLM)-based abstraction. A model can review thousands of charts in the time a person reviews a handful, and it applies the same instructions to every one. LLM-based abstraction is increasingly accurate and performs close to trained human abstractors for many variables (citation 1, citation 2, citation 3). However, LLMs have known weak spots, including variables that require clinical judgment in context, and dates, which are often implied rather than stated.

Moving from LLM to human abstraction multiplies the effort, cost and time involved, but for many variables it moves quality only a little. For other variables, the gap is larger and worth paying for.

Because no single abstraction method fits every variable, flexible approaches are needed to dial up quality where it matters.

Two steps to quality calibration

When deciding whether to invest more resources to increase quality, there are two steps.

Step one: Measure every option against a reliable reference standard in a test set

Measuring quality starts with a reference standard: a set of patient charts reviewed with the most rigorous process available and treated as the truth. Every abstraction option, including human abstraction, is then compared against it. This point is easy to miss. It's tempting to use human abstractors as the benchmark for an LLM, but people make mistakes too. Comparing two imperfect methods with each other can't tell you which one is right. However if a test set is abstracted independently by the LLM and humans, and disagreements are adjudicated, or a reliable external reference standard is available, performance between two abstraction approaches can be compared.

We measure both completeness and accuracy. Completeness asks whether a value was found at all: if a patient had a FibroScan, did we capture it? Accuracy asks whether the captured value is correct.

We also measure each field separately, because a single test has several fields and they can behave differently. An LLM might reliably pull a liver stiffness value from a report but miss the test date more often, when the date appears only in a header or has to be inferred from the visit.

Finally, test sets need to be large enough that a difference of a few percentage points reflects a real difference between methods rather than noise from a small sample. This approach builds on the VALID framework for validating LLM- and machine-learning-extracted real-world data.

Step two: Turn the dial where it counts

Once you know how each abstraction option performs, particularly their weaknesses, additional and combinations of approaches can be deployed to greatly enhance the quality, including:

• By field: If an LLM captures FibroScan test results accurately but misses some dates, human review can target the dates alone.

• By study stage: LLM-abstracted data can screen a large population to find eligible patients (e.g., identifying patients with more than one FibroScan test), then the much smaller cohort can get human confirmation where it matters for the analysis.

• By record: Only the charts where the LLM’s answer is missing or is internally inconsistent with other data points get routed to a person.

• Statistically: A validation subsample, where both LLM output and the reference standard are available, lets analysts estimate how much measurement error could affect the results and adjust for it, without re-abstracting the full cohort.

None of this works if each new approach means rebuilding the dataset. At Pedestal Health, each abstracted value carries a record of how it was obtained: structured, LLM-abstracted, human-abstracted or adjudicated. A study can specify which source to use for each variable and each stage, and change that choice as the study's needs become clear.

Case study: FibroScan results at scale

For our study being presented at The Liver Meeting 2026, we used LLM-based abstraction to quickly extract CAP values across more than 3,000 patients in a U.S. MASLD cohort to help unlock meaningful research about CAP score variation across cardiometabolic and racial subgroups. The LLM matched human performance closely on CAP values (within 4 points of human accuracy), which made the study practical and allowed us to tackle the research question at hand.

However, date completeness lagged (84% for LLM vs. 94% for human abstraction), which is where the dial comes in. A study that only needs to know whether a patient had a FibroScan, or what the CAP and LSM result was, may be well served by LLM abstraction alone. A study that needs to know when, to set fibrosis stage at the start of follow-up or to measure change between tests, loses every patient whose test date is missing. Rather than re-abstracting every test by hand, an efficient design keeps the LLM output as the default and sends only the 16% of tests without a date to human abstractors. That concentrates human effort on about one test in six and could recover much of the gap. It won't recover all of it, since some charts never record a date.

Not every variable suits LLM abstraction as well as FibroScan information, which is documented in a fairly predictable format. Variables that are documented inconsistently across notes, described in degrees or in the negative, or that serve as study outcomes often warrant a higher setting on the dial, such as human review of LLM output with adjudication of disagreements.

Considerations for study teams

Quality shouldn't be a black box, and it shouldn't be one-size-fits-all. Three principles follow:

1. Ask about quality. When evaluating a dataset, ask how each key variable in your study was captured, validated and against what reference.

2. Match the investment to the variable's role. Variables that select patients, define outcomes and anchor time each carry different risks, and they warrant different levels of review.

3. Spend human review where it can improve completeness and accuracy. Targeting review to specific fields, study stages or uncertain records can capture much of the benefit of full manual abstraction at a fraction of the cost and time.

Quality should be measured, visible and tunable. That's the approach we take at Pedestal Health: understand how every abstraction option performs, then turn the dial where each study needs it.

Learn more

Melissa Estevez will present this work, “CAP Score Variation Across Cardiometabolic and Racial Subgroups in a U.S. MASLD Cohort: Does Race Modify the Metabolic Liver Disease Burden?” at The Liver Meeting 2026 by AASLD. Visit Pedestal Health at Booth 1968 to discuss fit-for-purpose variable strategies.

The clinically meaningful variables behind this work are expert-defined and AI-abstracted in Pedestal Health’s Hepatology Core Data, a licensable, analysis-ready, longitudinal real-world dataset.