Papers with IRT
Item Response Theory for Natural Language Processing (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial introduces the wider NLP community to Item Response Theory (IRT) existing software for fitting IRT models is limited by human-data sized constraints. |
| Approach: | They will introduce IRT and the mathematical foundations which make IRT models. |
| Outcome: | This tutorial aims to introduce the wider NLP community to Item Response Theory and show its benefits for a number of NLP tasks. |
Predicting Difficulty and Discrimination of Natural Language Questions (2022.acl-short)
Copied to clipboard
| Challenge: | Item Response Theory (IRT) has been used to numerically characterize question difficulty and discrimination for human subjects in domains including cognitive psychology and education. |
| Approach: | They explore the relationship between difficulty and discrimination in question-answering contexts by using IRT to characterize item difficulty and item discrimination. |
| Outcome: | The proposed models can predict difficulty and discrimination parameters for new questions and explain them with features of questions, answers, and associated contexts. |
MAQuA: Multi-outcome Adaptive Question-Asking for Mental Health using Item Response Theory (2026.eacl-long)
Copied to clipboard
Vasudha Varadarajan, Hui Xu, Rebecca Astrid Böhme, Mariam Marlen Mirström, Sverker Sikström, H. Andrew Schwartz
| Challenge: | Evaluations of large language models (LLMs) indicate that such assessments are inconsistent and in many cases less accurate than dedicated condition-specific models with established psychometric validity. |
| Approach: | They propose a multi-outcome modeling and adaptive question-asking framework for simultaneous, multidimensional mental health screening that integrates language responses with item response theory and factor analysis. |
| Outcome: | Empirical results show that MAQuA reduces the number of assessment questions required for score stabilization by 50–87% compared to random ordering. |
Revisiting Generalization Across Difficulty Levels: It’s Not So Easy (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing research is mixed regarding whether training on easier or harder data leads to better results. |
| Approach: | They examine how well large language models generalize across different task difficulties by using a large dataset and a well-established difficulty metric. |
| Outcome: | The results show that training on hard data can't achieve consistent improvements across the full range of difficulties. |
Investigating the Zone of Proximal Development of Language Models for In-Context Learning (2025.findings-naacl)
Copied to clipboard
| Challenge: | In-context learning is a dynamic and progressive process where learners integrate new information into their knowledge base through interactions with the environment. |
| Approach: | They propose a learning analytics framework to analyze the in-context learning behavior of large language models (LLMs) through the lens of the Zone of Proximal Development (ZPD), an established theory in educational psychology. |
| Outcome: | The proposed framework improves inference and fine-tuning scenarios by selectively applying it to queries that are most likely to benefit from demonstrations. |
Identifying Annotator Bias: A new IRT-based method for bias identification (2020.coling-main)
Copied to clipboard
| Challenge: | An important factor that can affect the Inter Annotator Agreement (IAA) is the presence of annotator bias. |
| Approach: | They propose a new interpretation and application of the Item Response Theory to detect annotators' bias and characterise annotation disagreement. |
| Outcome: | The proposed method can be used to spot outliers, improve annotation guidelines and provide a better picture of the annotation reliability. |
Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds (D19-1)
Copied to clipboard
| Challenge: | Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. |
| Approach: | They propose to use IRT models generated from artificial crowds of DNNs to learn IRT. |
| Outcome: | The proposed model learning method outperforms baseline methods for two NLP tasks. |
Learning to Ask Conversational Questions by Optimizing Levenshtein Distance (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for estimating maximum likelihood are limited by easily learned tokens . Existing systems that generate questions based on dialogue context are limited in their ability to learn tokens. |
| Approach: | They propose a framework that optimizes the minimum Levenshtein distance through explicit editing actions. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on two benchmark datasets and generalizes well on unseen data. |
IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models have demonstrated exceptional performance across a wide range of tasks . however, selecting the optimal LLM to respond to a user query often necessitates a delicate balance between performance and cost. |
| Approach: | They propose a multi-LLM routing framework that efficiently routes user queries to the most suitable LLM. |
| Outcome: | The proposed framework outperforms baseline methods in terms of effectiveness and interpretability. |
SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Traditionally, estimating item difficulties requires real students to respond to items . a cold-start approach cannot be applied to previously unseen items either . |
| Approach: | They propose a method for aligning simulated students with instructed ability to predict difficulty of open-ended items. |
| Outcome: | The proposed method outperforms existing methods on two real-world student responses. |
AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing audio question answering benchmarks emphasize sound event classification or caption-grounded queries. |
| Approach: | They propose a large-scale, real-world audio question answering benchmark to evaluate audio reasoning beyond surface-level acoustic recognition. |
| Outcome: | The proposed model achieves 32.13% accuracy while demonstrating comprehension of audio . state-of-the-art models perform poorly, with average accuracy below 8.86%. |
Ranking Reasoning LLMs under Test-Time Scaling (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as general-purpose reasoning systems for tasks such as programming and mathematical problem solving. |
| Approach: | They formalize dense benchmark ranking under test-time scaling and introduce a library that implements statistical ranking methods such as paired-comparison models, item response theory, voting rules, graph- and spectral-based methods. |
| Outcome: | The proposed method is based on paired-comparison models, item response theory (IRT) models, voting rules, graph- and spectral-based methods. |
Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations (2026.findings-acl)
Copied to clipboard
| Challenge: | Standardized math assessments require expensive human pilot studies to establish the difficulty of test items. |
| Approach: | They propose to use large language models to model difficulty of multiple-choice math questions for real-world students. |
| Outcome: | The proposed model predicts difficulty of multiple-choice math questions for students . correlations between model and real-world difficulty are high, the authors show . |