SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Traditionally, estimating item difficulties requires real students to respond to items . a cold-start approach cannot be applied to previously unseen items either . |
| Approach: | They propose a method for aligning simulated students with instructed ability to predict difficulty of open-ended items. |
| Outcome: | The proposed method outperforms existing methods on two real-world student responses. |
Similar Papers
Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction (2026.findings-acl)
Copied to clipboard
| Challenge: | Accurate estimation of item (question or task) difficulty suffers from the cold start problem. |
| Approach: | They propose to use large-scale empirical analysis to examine human-AI Difficulty Alignment . they find that models struggle to simulate the capability limitations of students . |
| Outcome: | The proposed model size is not reliably helpful for human-AI alignment . high performance often impedes accurate difficulty estimation, the authors say . |
Predicting Difficulty and Discrimination of Natural Language Questions (2022.acl-short)
Copied to clipboard
| Challenge: | Item Response Theory (IRT) has been used to numerically characterize question difficulty and discrimination for human subjects in domains including cognitive psychology and education. |
| Approach: | They explore the relationship between difficulty and discrimination in question-answering contexts by using IRT to characterize item difficulty and item discrimination. |
| Outcome: | The proposed models can predict difficulty and discrimination parameters for new questions and explain them with features of questions, answers, and associated contexts. |
Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations (2026.findings-acl)
Copied to clipboard
| Challenge: | Standardized math assessments require expensive human pilot studies to establish the difficulty of test items. |
| Approach: | They propose to use large language models to model difficulty of multiple-choice math questions for real-world students. |
| Outcome: | The proposed model predicts difficulty of multiple-choice math questions for students . correlations between model and real-world difficulty are high, the authors show . |
Large Language Models are Students at Various Levels: Zero-shot Question Difficulty Estimation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advancements in educational platforms have emphasized the importance of personalized education. |
| Approach: | They propose a framework that utilizes large language models to represent students at various levels to estimate question difficulty with and without student question-solving records. |
| Outcome: | The proposed framework outperforms baseline models on the DBE-KT22 and ASSISTMents 2005–2006 benchmarks and shows a high correlation with the regressed IRT curve. |
Revisiting Generalization Across Difficulty Levels: It’s Not So Easy (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing research is mixed regarding whether training on easier or harder data leads to better results. |
| Approach: | They examine how well large language models generalize across different task difficulties by using a large dataset and a well-established difficulty metric. |
| Outcome: | The results show that training on hard data can't achieve consistent improvements across the full range of difficulties. |
Can Model Uncertainty Function as a Proxy for Multiple-Choice Question Item Difficulty? (2025.coling-main)
Copied to clipboard
| Challenge: | Supervised approaches to difficulty estimation have yielded mixed results . generative large models are seen as a weakness when answering questions . |
| Approach: | They exploit generative large models to explore correlations between two different metrics of uncertainty, and the actual student response distribution. |
| Outcome: | The proposed model uncertainty is different in the case of correct vs wrong answers and the student response distribution is different. |
Item Response Theory for Natural Language Processing (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial introduces the wider NLP community to Item Response Theory (IRT) existing software for fitting IRT models is limited by human-data sized constraints. |
| Approach: | They will introduce IRT and the mathematical foundations which make IRT models. |
| Outcome: | This tutorial aims to introduce the wider NLP community to Item Response Theory and show its benefits for a number of NLP tasks. |
Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in math reasoning problemsolving, text generation, summarization, creative writing, among other tasks. |
| Approach: | They evaluate Direct Preference Optimization and its variants for aligning Large Language Models with human preferences. |
| Outcome: | The proposed alignment methods achieve near-optimal performance even with smaller subsets of training data. |
Generating and Evaluating Tests for K-12 Students with Language Model Simulations: A Case Study on Sentence Reading Efficiency (2023.emnlp-main)
Copied to clipboard
| Challenge: | Developing an educational test can be expensive and time-consuming, as each item must be written by experts and then evaluated by collecting hundreds of student responses. |
| Approach: | They propose to fine-tune large language models to simulate how previous students would have responded to unseen items to generate high-quality parallel tests. |
| Outcome: | The proposed test forms are designed to be content-equivalent and produce identical individual scores as the original test form. |
One LLM Does Not Simulate All Students: Ability-Aware Student Simulation via Cognitive Diagnosis Guided LLM Assignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods rely on a single high-capacity LLM to represent an entire population of diverse learners. |
| Approach: | They propose an ability-aware student simulation framework that matches students with appropriate LLM backbones through cognitive alignment. |
| Outcome: | The proposed framework significantly reduces simulation bias and outperforms single-model baselines across the entire proficiency spectrum. |