Predicting Difficulty and Discrimination of Natural Language Questions (2022.acl-short)
Copied to clipboard
| Challenge: | Item Response Theory (IRT) has been used to numerically characterize question difficulty and discrimination for human subjects in domains including cognitive psychology and education. |
| Approach: | They explore the relationship between difficulty and discrimination in question-answering contexts by using IRT to characterize item difficulty and item discrimination. |
| Outcome: | The proposed models can predict difficulty and discrimination parameters for new questions and explain them with features of questions, answers, and associated contexts. |
Similar Papers
Item Response Theory for Natural Language Processing (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial introduces the wider NLP community to Item Response Theory (IRT) existing software for fitting IRT models is limited by human-data sized constraints. |
| Approach: | They will introduce IRT and the mathematical foundations which make IRT models. |
| Outcome: | This tutorial aims to introduce the wider NLP community to Item Response Theory and show its benefits for a number of NLP tasks. |
SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Traditionally, estimating item difficulties requires real students to respond to items . a cold-start approach cannot be applied to previously unseen items either . |
| Approach: | They propose a method for aligning simulated students with instructed ability to predict difficulty of open-ended items. |
| Outcome: | The proposed method outperforms existing methods on two real-world student responses. |
Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds (D19-1)
Copied to clipboard
| Challenge: | Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. |
| Approach: | They propose to use IRT models generated from artificial crowds of DNNs to learn IRT. |
| Outcome: | The proposed model learning method outperforms baseline methods for two NLP tasks. |
Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations (2026.findings-acl)
Copied to clipboard
| Challenge: | Standardized math assessments require expensive human pilot studies to establish the difficulty of test items. |
| Approach: | They propose to use large language models to model difficulty of multiple-choice math questions for real-world students. |
| Outcome: | The proposed model predicts difficulty of multiple-choice math questions for students . correlations between model and real-world difficulty are high, the authors show . |
IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models have demonstrated exceptional performance across a wide range of tasks . however, selecting the optimal LLM to respond to a user query often necessitates a delicate balance between performance and cost. |
| Approach: | They propose a multi-LLM routing framework that efficiently routes user queries to the most suitable LLM. |
| Outcome: | The proposed framework outperforms baseline methods in terms of effectiveness and interpretability. |
Comparing Test Sets with Item Response Theory (2021.acl-long)
Copied to clipboard
Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman
| Challenge: | Recent results from large pretrained models show that many datasets are saturated and unlikely to detect further progress. |
| Approach: | They evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. |
| Outcome: | The proposed datasets are saturated and unlikely to detect future improvements. |
Revisiting Generalization Across Difficulty Levels: It’s Not So Easy (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing research is mixed regarding whether training on easier or harder data leads to better results. |
| Approach: | They examine how well large language models generalize across different task difficulties by using a large dataset and a well-established difficulty metric. |
| Outcome: | The results show that training on hard data can't achieve consistent improvements across the full range of difficulties. |
Difficulty-Focused Contrastive Learning for Knowledge Tracing with a Large Language Model-Based Difficulty Prediction (2024.lrec-main)
Copied to clipboard
Unggi Lee, Sungjun Yoon, Joon Seo Yun, Kyoungsoo Park, YoungHoon Jung, Damji Stratton, Hyeoncheol Kim
| Challenge: | Existing studies have focused on incorporating the difficulty information into knowledge tracing models, but few studies have explored the potential of difficulty estimation. |
| Approach: | They propose a difficulty-centered contrastive learning method and a Large Language Model-based framework for difficulty prediction to improve the performance of knowledge tracing models. |
| Outcome: | The proposed methods demonstrate enhanced performance of knowledge tracing models while ignoring the complex relationship between language and difficulty. |
Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction (2026.findings-acl)
Copied to clipboard
| Challenge: | Accurate estimation of item (question or task) difficulty suffers from the cold start problem. |
| Approach: | They propose to use large-scale empirical analysis to examine human-AI Difficulty Alignment . they find that models struggle to simulate the capability limitations of students . |
| Outcome: | The proposed model size is not reliably helpful for human-AI alignment . high performance often impedes accurate difficulty estimation, the authors say . |
Statistically Profiling Biases in Natural Language Reasoning Datasets and Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate NLP models' weaknesses are limited by “hypothesis-only” tests and CheckLists. |
| Approach: | They propose a lightweight general statistical profiling framework that automatically identifies potential biases in multiple-choice NLU datasets without requiring additional test cases. |
| Outcome: | The proposed framework assesses the extent to which models exploit these biases through black-box testing, confirming prior findings and revealing new insights. |