Generating Multiple-choice Questions for Medical Question Answering with Distractors and Cue-masking (2024.lrec-main)
Copied to clipboard
| Challenge: | Medical multiple-choice question answering (MCQA) requires high accuracy to be useful in practice. |
| Approach: | They propose to focus masked language modeling on disease name prediction when using medical encyclopedic paragraphs as input. |
| Outcome: | The proposed model outperforms the masked language model on disease name prediction and masks the cues to the answers. |
Similar Papers
Clues Before Answers: Generation-Enhanced Multiple-Choice QA (2022.naacl-main)
Copied to clipboard
| Challenge: | Multiple-choice question answering (MCQA) uses text-to-text framework . but, there is an under-utilization of the decoder and knowledge that can be decoded . |
| Approach: | They propose a generative multiple-choice question answering model which generates a clue from the question and leverages it to enhance a reader for MCQA. |
| Outcome: | The proposed model outperforms text-to-text models on multiple MCQA datasets. |
DisGeM: Distractor Generation for Multiple Choice Questions with Span Masking (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Multiple-choice cloze tests are a prevalent form of assessment that evaluates students' comprehension and inference abilities. |
| Approach: | They propose a framework for distractor generation using readily available pre-trained language models . human evaluations confirm that their approach produces more effective distractors . |
| Outcome: | The proposed framework outperforms existing methods without training or fine-tuning human evaluations confirm it. |
When Models Decide and When They Bind: A Two-Stage Computation for Multiple-Choice Question Answering (2026.findings-acl)
Copied to clipboard
| Challenge: | Multiple-choice question answering (MCQA) is easy to evaluate but adds a meta-task . prior work has shown that language models exhibit selection biases for particular option identifiers such as the label "A" |
| Approach: | They find that option-boundary residual states contain strong linearly decodable signals . winning content position becomes decoded after final option is processed . |
| Outcome: | The proposed model solves the problem and outputs the symbol that represents the answer. |
Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above (2025.acl-long)
Copied to clipboard
| Challenge: | Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing. |
| Approach: | They argue for a reform of multiple choice question answering (MCQA) they argue for more generative formats based on human testing . |
| Outcome: | The proposed reforms improve the quality of MCQA, the authors argue . they show that even when MCQ is a useful format, its datasets suffer from leakage, unanswerability, shortcuts and saturation. |
To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering (2024.acl-long)
Copied to clipboard
| Challenge: | Medical open-domain question answering requires substantial access to specialized knowledge. |
| Approach: | They propose a framework that generates multiple-choice questions from a set of open-book parameters and a small-scale reader that can outcompete closed-book questions by 706x using fewer parameters. |
| Outcome: | The proposed framework outperforms closed-book models on MedQA-USMLE, MedMCQA, and MMLU while using up to 706x fewer parameters. |
Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are often evaluated using multiple-choice questions (MCQs) modeled on exams like the USMLE. |
| Approach: | They created a fictional medical benchmark centered on an imaginary organ, the Glianorex, to separate memorized knowledge from reasoning ability. |
| Outcome: | The proposed model outperforms base models in English but not in French. |
Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? (2024.acl-long)
Copied to clipboard
| Challenge: | Multiple-choice question answering (MCQA) is often used to evaluate large language models . a recent study found that LLMs perform MCQA with choices-only prompts . |
| Approach: | They investigate whether LLMs can perform multiple-choice question answering (MCQA) with choices-only prompts . they find no evidence that the choices- only accuracy stems from memorization alone . |
| Outcome: | The results show that LLMs perform MCQA with choices-only prompts with 0.33 accuracy gain. |
Can We Learn Question, Answer, and Distractors All from an Image? A New Task for Multiple-choice Visual Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies focus on generating QADs from image and question, but a novel task is needed to generate meaningful questions, correct answers, and challenging distractors. |
| Approach: | They propose a task to generate QADs from images and encode images together . they use contrastive learning to ensure consistency of QAD generated and tested . |
| Outcome: | Empirical evaluations on the benchmark dataset validate the performance of the proposed task. |
ReMedQA: Are We Done With Medical Multiple-Choice Benchmarks? (2026.eacl-long)
Copied to clipboard
| Challenge: | Multiple-choice question answering (MCQA) benchmarks show near-human accuracy . but a single accuracy score is a poor proxy for competence . |
| Approach: | They propose a medical multiple-choice question answering (MCQA) benchmark that augments three standard medical MCQA datasets with open-ended answers and systematically perturbed options. |
| Outcome: | The proposed benchmarks show that high MCQA accuracy masks low reliability . MCQ is the dominant paradigm for assessing medical knowledge in large language models . |
Multi-modal Concept Alignment Pre-training for Generative Medical Visual Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Medical Visual Question Answering (Med-VQA) aims to provide accurate answers to questions regarding medical images, a task particularly challenging for open-ended questions. |
| Approach: | They propose a multi-modal concept alignment pre-training approach for generative Med-VQA that leverages a knowledge graph sourced from medical image-caption datasets and the Unified Medical Language System. |
| Outcome: | The proposed approach significantly outperforms existing methods on a set of benchmark datasets and shows high efficiency and knowledge-image alignment capability. |