TEEMIL : Towards Educational MCQ Difficulty Estimation in Indic Languages (2025.coling-main)
Copied to clipboard
| Challenge: | Traditionally, educators manually create and calibrate MCQs, a process that is time-consuming and subjective. |
| Approach: | They propose to use TEEMIL-H and TEIMEL-K to create a dataset with manually annotated difficulty labels for MCQs in Hindi and Kannada. |
| Outcome: | The proposed datasets contain 4689 and 4215 MCQs with manually annotated difficulty labels. |
Similar Papers
Can Multiple-choice Questions Really Be Useful in Detecting the Abilities of LLMs? (2024.lrec-main)
Copied to clipboard
| Challenge: | Multiple-choice questions (MCQs) are widely used in the evaluation of large language models (LLMs) however, there are concerns about whether MCQ can truly measure LLM’s capabilities. |
| Approach: | They propose to use multiple choice questions to evaluate large language models (LLMs) to assess their capabilities. |
| Outcome: | The proposed methods show that MCQs are less reliable than LFGQs in terms of expected calibration error. |
Exploring Automated Distractor Generation for Math Multiple-choice Questions via Large Language Models (2024.findings-naacl)
Copied to clipboard
Wanyong Feng, Jaewook Lee, Hunter McNichols, Alexander Scarlatos, Digory Smith, Simon Woodhead, Nancy Ornelas, Andrew Lan
| Challenge: | Multiple-choice questions (MCQs) are easy to administer and grade . but crafting high-quality distractors remains labor-intensive and limited scalability . |
| Approach: | They propose to automate the generation of distractors in math MCQs by using large language models to generate distractors. |
| Outcome: | The proposed methods can generate valid distractors, but they are less adept at anticipating common errors or misconceptions among real students. |
Revisiting the Self-Consistency Challenges in Multi-Choice Question Formats for Large Language Model Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Multi-choice questions (MCQs) are a common method for assessing the world knowledge of large language models. |
| Approach: | They propose three knowledge-equivalent question variants to assess LLMs' world knowledge . they propose option position shuffle, option label replacement, and conversion to a True/False format . |
| Outcome: | The proposed questions are shuffle, label replacement, and True/False format. |
The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for difficulty estimation rely on repeated response sampling, auxiliary models, or fine-tuning the target model itself. |
| Approach: | They propose a method that leverages only the hidden representations produced by large language models. |
| Outcome: | The proposed method outperforms baselines in difficulty estimation on textual and multimodal tasks and improves adaptive reasoning strategies with fewer generated tokens. |
Revisiting Evaluation of Question Answering Systems in Low-Resource Indic Languages: Bridging Human and Metric Alignment (2026.acl-short)
Copied to clipboard
| Challenge: | Evaluating Question Answering systems in low-resource Indic languages remains challenging due to the scarcity of annotated data and the lack of reliable evaluation metrics. |
| Approach: | They propose a language-based multi-aspect evaluation framework for question answering systems . the framework integrates semantic similarity, factual completeness, numerical accuracy and contextual relevance . |
| Outcome: | The proposed metric is evaluated across eight Indic-language QA tasks using multiple LLMs . Across all settings, it shows stronger agreement with human evaluation . |
INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages (2025.findings-naacl)
Copied to clipboard
Abhishek Kumar Singh, Vishwajeet Kumar, Rudra Murthy, Jaydeep Sen, Ashish Mittal, Ganesh Ramakrishnan
| Challenge: | Large Language Models perform well on unseen tasks in English, but their abilities in non-English languages are less explored due to limited benchmarks and training data. |
| Approach: | They propose to release a large dataset for context-grounded question answering in 11 major Indian languages. |
| Outcome: | The Indic-QA Benchmark compared large datasets of large LLMs on extractive and abstractive tasks in 11 major Indian languages. |
Large Language Models are Students at Various Levels: Zero-shot Question Difficulty Estimation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advancements in educational platforms have emphasized the importance of personalized education. |
| Approach: | They propose a framework that utilizes large language models to represent students at various levels to estimate question difficulty with and without student question-solving records. |
| Outcome: | The proposed framework outperforms baseline models on the DBE-KT22 and ASSISTMents 2005–2006 benchmarks and shows a high correlation with the regressed IRT curve. |
Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations (2026.findings-acl)
Copied to clipboard
| Challenge: | Standardized math assessments require expensive human pilot studies to establish the difficulty of test items. |
| Approach: | They propose to use large language models to model difficulty of multiple-choice math questions for real-world students. |
| Outcome: | The proposed model predicts difficulty of multiple-choice math questions for students . correlations between model and real-world difficulty are high, the authors show . |
What Kind of Language Is Hard to Language-Model? (P19-1)
Copied to clipboard
| Challenge: | a recent study suggests that language models perform poorly across languages. |
| Approach: | They propose a model that fits a paired-sample multiplicative mixed-effects model to obtain language difficulty coefficients from at least-pairwise parallel corpora. |
| Outcome: | The proposed model is able to handle missing data and is aware of inter-sentence variation. |
Revisiting Generalization Across Difficulty Levels: It’s Not So Easy (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing research is mixed regarding whether training on easier or harder data leads to better results. |
| Approach: | They examine how well large language models generalize across different task difficulties by using a large dataset and a well-established difficulty metric. |
| Outcome: | The results show that training on hard data can't achieve consistent improvements across the full range of difficulties. |