Toward a Consensus Standard for Evaluating AI-Generated Health Summaries: A Conceptual Framework

The Problem

Large language models can now generate fluent, well-organized summaries of complex medical records. What we cannot yet do is say, with rigor, whether any given summary is good. There is no accepted gold standard for health summary quality, no consensus definition of what such a summary must contain, and no validated process for grading one at scale. Evaluation today largely rests on ad hoc clinician impressions — a method that is unscalable, unreproducible, and unsuited to the safety stakes involved.

The absence of a standard is not a measurement gap alone; it is a conceptual gap. A summary is not good or bad in the abstract. It is good or bad for a purpose and for a reader. The information a cardiologist needs before a clinic visit differs from what a primary care physician needs at an annual exam, which differs again from what a surgeon needs preoperatively. Any credible evaluation framework must accommodate this variation without fragmenting into unusable, per-user idiosyncrasy.

This document proposes a conceptual framework for constructing such a standard.

Core Premise: Evaluate Against Criteria, Not Reference Summaries

A common instinct is to build a corpus of "gold standard" summaries — expert-written references against which machine output is compared. For clinical summarization this approach fails on three counts. It does not scale: every patient's record would require its own hand-crafted reference. It does not generalize: two competent clinicians will write materially different summaries of the same chart, so no single reference is authoritative. And it measures the wrong thing: similarity to a reference text is a poor proxy for clinical adequacy.

The alternative is criterion-referenced evaluation. Rather than asking "does this summary resemble the one an expert would write?", we ask "does this summary satisfy an explicit, consensus-ratified set of quality criteria?" The gold standard, on this view, is not a corpus of summaries. It is a validated criteria set together with a validated grading process. That reframing makes the problem tractable, and it is the foundation of everything that follows.

Prior work provides a starting point rather than a blank page. Instruments such as the Physician Documentation Quality Instrument (PDQI-9) and its successor adapted for LLM output (PDSQI-9), along with recent evaluations of LLM clinical summarization organized around completeness, correctness, and conciseness, demonstrate that structured, rater-validated assessment of clinical text is feasible. What remains missing is a framework that binds such instruments to specialty-specific content requirements grounded in clinical consensus.

A Two-Layer Architecture for Quality Criteria

The framework separates quality into two layers, reflecting the distinction between properties every summary must have and content particular audiences require.

Layer one: universal criteria. These apply regardless of specialty or use case. They include faithfulness (every claim in the summary is traceable to the source record), absence of fabrication, correct attribution (findings assigned to the right encounter, problem, or provider), temporal accuracy (the chronology of the clinical course is preserved), internal consistency, organization, and concision. These criteria are audience-independent because they concern the summary's relationship to the truth of the record, not its usefulness to a particular reader.

Layer two: specialty-specific salience criteria. These define what a summary must contain to serve a given clinical audience — primary care, cardiology, surgery, and so on. For a cardiology-facing summary, salient elements might include ejection fraction, implanted devices, anticoagulation status, and recent interventions; for primary care, an accurate problem list, medication reconciliation, and preventive care status. The critical design choice is to tailor at the level of specialty rather than the individual user. Per-user tailoring is intractable and unvalidatable; specialty-level tailoring maps onto existing bodies of consensus — national practice guidelines, specialty society standards, and quality measure sets — that already encode professional agreement about what information matters for decision-making in each domain.

Manufacturing Consensus: The Role of Formal Methods

Deriving layer-two criteria directly from guidelines is necessary but not sufficient. Guidelines specify care standards, not documentation standards, and translating one into the other involves judgment. That judgment should not belong to any single research team.

The framework therefore calls for formal consensus methodology — a modified Delphi process or the RAND/UCLA appropriateness method — in which panels of specialty clinicians iteratively rate and refine candidate criteria until convergence. These methods exist precisely to produce defensible consensus where none previously existed, and they are what elevates a research team's rubric into a standard the field can adopt. The output of each panel is a ratified salience checklist for that specialty, versioned and revisable as guidelines evolve.

The Problem of Negative Information

A distinctive challenge in summary evaluation is information that is absent. Omission is the dominant and least detectable failure mode of machine-generated summaries: a fluent document that silently drops a critical laboratory trend or an active safety issue reads as complete while being dangerous.

The framework addresses this by distinguishing two kinds of absence. Pertinent negatives are clinically meaningful absences that a good summary should affirmatively state — no known drug allergies, a negative troponin in a chest pain workup, the absence of anticoagulation in a patient with atrial fibrillation. Silent omissions are salient facts present in the record but missing from the summary. The salience checklist operationalizes both: each required element for a specialty is scored as present, appropriately noted as absent, or silently missing, with silent omission treated as an error on par with fabrication. Evaluation frameworks that score only what a summary says, and never what it fails to say, are structurally blind to the most consequential failure class.

The Place of Human Review

Human judgment is indispensable — but at specific points, not everywhere. The framework assigns clinicians three roles.

First, standard-setting: clinicians constitute the consensus panels that ratify the criteria. Second, calibration: clinicians score a reference corpus of summaries against the finished rubric, with inter-rater reliability formally measured (weighted kappa or intraclass correlation). This step validates that the rubric is usable by humans at all, and it produces the benchmark against which any automated grader must be tested. Third, audit and adjudication: humans review disagreements, sample scored output on an ongoing basis, and monitor for drift as models and guidelines change.

Between these anchor points, evaluation can and should be automated. Rubric-based scoring by language models ("LLM-as-judge") has shown adequate agreement with expert raters when — and only when — it is validated against a human-scored calibration set, with agreement reported by specialty and by error type. Faithfulness checking is particularly amenable to automation when summaries are required to cite claim-level provenance back to the source record, allowing programmatic verification. Where source data exists in structured form, salience checking is likewise partially machine-verifiable: one can ask directly whether the record contained a given element and whether the summary carried it forward.

The Evaluation Pipeline, Assembled

The full framework proceeds in five stages: (1) adopt and adapt validated universal criteria from existing documentation-quality instruments; (2) derive specialty salience checklists from national consensus guidelines and ratify them through formal consensus panels; (3) build a human-scored calibration corpus with demonstrated inter-rater reliability; (4) validate automated graders against that corpus before deployment at scale; and (5) maintain human audit, adjudication, and periodic re-ratification as guidelines and models evolve.

The resulting gold standard is thus a composite artifact: the two-layer rubric, the consensus provenance behind it, the reliability statistics of human raters using it, and the validation statistics of any automated grader standing in for them. Each component is publishable, reproducible, and contestable — which is what distinguishes a standard from an opinion.

Implications

For health systems, this framework offers a path to deploying AI summarization with measurable, audience-appropriate quality assurance rather than anecdotal confidence. For researchers, it defines a shared measurement substrate that makes summarization studies comparable across models and institutions. And for the broader field, it demonstrates a general pattern: where no gold standard exists, the productive move is not to search for one but to construct one — by binding validated measurement instruments to domain consensus through formal methods, and by placing human expertise where it is irreplaceable rather than everywhere it is expensive.

Comments and collaboration inquiries are welcome.