Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation (2022.emnlp-industry)
Copied to clipboard
Aleksandar Savkov, Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Anya Belz, Ehud Reiter
| Challenge: | Existing studies have shown that note generation is difficult due to subjective nature of many aspects of output quality. |
| Approach: | They propose a protocol that aims to increase objectivity by grounding evaluations in Consultation Checklists, which are created in a preliminary step and then used as a common point of reference during quality assessment. |
| Outcome: | The proposed protocol shows that the evaluations produced in the study are more objective than the original human note. |
Similar Papers
Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation (2022.acl-long)
Copied to clipboard
Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Damir Juric, Jack Flann, Ehud Reiter, Anya Belz, Aleksandar Savkov
| Challenge: | Recent studies suggest that note generation systems can be used to generate clinical consultation notes from the verbatim transcript of the consultation. |
| Approach: | They propose to use machine learning to generate consultation notes from the verbatim transcript of the consultation to evaluate their effectiveness. |
| Outcome: | The proposed model performs better than common model-based metrics like BertScore and is open-sourced. |
From Feedback to Checklists: Grounded Evaluation of AI-Generated Clinical Notes (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing automated metrics fail to align with real-world physician preferences. |
| Approach: | They propose a pipeline that distills real user feedback into structured checklists for note evaluation that are interpretable, grounded in human feedback, and enforceable by LLM-based evaluators. |
| Outcome: | The proposed checklist outperforms baseline evaluations in coverage, diversity, and predictive power for human ratings. |
An Investigation of Evaluation Methods in Automatic Medical Note Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that doctors can save significant amounts of time when using automatic note generation. |
| Approach: | They propose task-specific metrics for automatic note generation from medical conversation summarization and generation, including knowledge-graph embedding-based metrics, customized model-based measures with domain-specific weights, and ensemble metrics. |
| Outcome: | The proposed evaluation metrics are compared to existing models and can have different behaviors on different types of clinical notes datasets. |
CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation protocols for text generation suffer from rating inconsistencies . lexical overlap-based metrics align poorly with human judgments . |
| Approach: | They propose a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. |
| Outcome: | The proposed framework improves rating reliability by decomposing binary questions . it improves agreement across evaluator models by 0.45 and reduces score variance . human evaluation remains the gold standard, but it #, Equal contribution. |
User-Driven Research of Medical Note Generation Software (2022.naacl-main)
Copied to clipboard
Tom Knoll, Francesco Moramarco, Alex Papadopoulos Korfiatis, Rachel Young, Claudia Ruffini, Mark Perera, Christian Perstl, Ehud Reiter, Anya Belz, Aleksandar Savkov
| Challenge: | Existing studies on how NLP systems could be used in clinical practice focus on technical difficulties and usability challenges involved in implementing them. |
| Approach: | They propose to use Speech Recognition to transcribe the audio of a medical consultation and then to train sequence-to-sequence models to summarise the transcript into a consultation note. |
| Outcome: | The proposed system generates notes in real time during a doctor-patient consultation and is able to capture the salient points of a consultation . the proposed system is based on three rounds of user studies in a live telehealth clinic and identifies a number of clinical use cases that could prove challenging for the system. |
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks? (2025.emnlp-main)
Copied to clipboard
| Challenge: | evaluators using large language models face ambiguous criteria and inconsistent evaluations. |
| Approach: | They investigate whether checklists should be used for all questions or selectively . they generate checklists using six methods and evaluate their effectiveness across eight models . |
| Outcome: | The proposed method improves evaluation performance in pairwise comparisons while ignoring human-written criteria. |
Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency? (2024.findings-naacl)
Copied to clipboard
| Challenge: | a clinical note is a document that documents a doctor's interaction with a patient . authors show that LLMs can be used to measure quality indicators . |
| Approach: | They analyze two different approaches to generate different sections of a SOAP note . they use PEGASUS-X Transformer models to examine note consistency . |
| Outcome: | The proposed approach leads to similar ROUGE values and no difference in Factuality metric . human reviewers perform the same tasks with roughly the same agreement as the LLMs . |
Annotate the Way You Think: An Incremental Note Generation Framework for the Summarization of Medical Conversations (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets for summarization of medical conversations are limited to conversation-summary pairs . a novel annotation framework is proposed to capture the summarizing process via an annotation task . |
| Approach: | They propose an incremental note generation framework that captures the human summarization process via an annotation task by instructing annotators to first incrementally create a draft note and polish it into a reference note. |
| Outcome: | The proposed framework shows that the human summarization process is much more efficient and accurate than the current method. |
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation (2022.emnlp-main)
Copied to clipboard
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, Daniel Weld
| Challenge: | Effective human evaluation of text generation tasks remains an important, open area for research. |
| Approach: | They propose a system for running standardized human evaluations across different generation tasks. |
| Outcome: | The proposed system produces standardized human evaluations across tasks . it crowdsources predictions and ranks systems on leaderboards . the proposed system is not reproducible over time and different annotator populations . |
Correction of Errors in Preference Ratings from Automated Metrics for Text Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods are over-confident in assigning significant differences between systems . Currently, the most reliable evaluation methods for text generation are human-based evaluations. |
| Approach: | They propose to combine human ratings with automated ratings to reduce the amount of human ratings needed to arrive at robust results. |
| Outcome: | The proposed evaluation protocol reduces the amount of human ratings by 50% while yielding the same evaluation outcome as the pure human evaluation in 95% of cases. |