Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds (D19-1)
Copied to clipboard
| Challenge: | Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. |
| Approach: | They propose to use IRT models generated from artificial crowds of DNNs to learn IRT. |
| Outcome: | The proposed model learning method outperforms baseline methods for two NLP tasks. |
Similar Papers
Item Response Theory for Natural Language Processing (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial introduces the wider NLP community to Item Response Theory (IRT) existing software for fitting IRT models is limited by human-data sized constraints. |
| Approach: | They will introduce IRT and the mathematical foundations which make IRT models. |
| Outcome: | This tutorial aims to introduce the wider NLP community to Item Response Theory and show its benefits for a number of NLP tasks. |
Predicting Difficulty and Discrimination of Natural Language Questions (2022.acl-short)
Copied to clipboard
| Challenge: | Item Response Theory (IRT) has been used to numerically characterize question difficulty and discrimination for human subjects in domains including cognitive psychology and education. |
| Approach: | They explore the relationship between difficulty and discrimination in question-answering contexts by using IRT to characterize item difficulty and item discrimination. |
| Outcome: | The proposed models can predict difficulty and discrimination parameters for new questions and explain them with features of questions, answers, and associated contexts. |
Comparing Test Sets with Item Response Theory (2021.acl-long)
Copied to clipboard
Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman
| Challenge: | Recent results from large pretrained models show that many datasets are saturated and unlikely to detect further progress. |
| Approach: | They evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. |
| Outcome: | The proposed datasets are saturated and unlikely to detect future improvements. |
Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on how to encode compositional task structure has been limited by semantic parsing and multihop reasoning for the purpose of Q&A. |
| Approach: | They propose an approach to decomposing a target task into component tasks and fine-tuning smaller LMs on a curriculum of such component tasks. |
| Outcome: | The proposed approach outperforms end-to-end learning even with equal data, and gets better as more component tasks are modeled. |
A Method for Building a Commonsense Inference Dataset based on Basic Events (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to acquire commonsense are limited by the general-purpose language models. |
| Approach: | They propose a method for building a commonsense inference dataset using crowdsourcing and automatic extraction from a corpus. |
| Outcome: | The proposed method can solve 104k commonsense inference problems in a Japanese corpus with high accuracy, but low bias. |
Adaptive Reinforcement Tuning Language Models as Hard Data Generators for Sentence Representation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods use contrastive learning (CL) to learn effective sentence representations, but require extensive human annotation. |
| Approach: | They propose a reinforcement learning approach for fine-tuning small-parameter LLMs to generate high-quality hard contrastive data without human feedback. |
| Outcome: | The proposed method achieves state-of-the-art on seven semantic text similarity tasks. |
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the importance of datasets for natural language understanding, there has been little attention on crowdsourcing methods for collecting datasets. |
| Approach: | They compare the effectiveness of crowdsourcing methods for boosting NLU example difficulty with training crowdworkers instead of expert judgments. |
| Outcome: | The proposed method is ineffective for boosting NLU example difficulty, but it is not effective for training crowdworkers and qualifying workers based on expert judgments. |
HellaSwag: Can a Machine Really Finish Your Sentence? (P19-1)
Copied to clipboard
| Challenge: | Existing commonsense models struggle to perform inferences that are trivial for humans, but are often misclassified by state-of-the-art models. |
| Approach: | They propose a dataset that is adversarial to state-of-the-art commonsense reasoning and use it to build a model that is surprisingly robust. |
| Outcome: | The proposed dataset is compared with existing models and scaled up towards a critical 'Goldilocks zone' wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. |
SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Traditionally, estimating item difficulties requires real students to respond to items . a cold-start approach cannot be applied to previously unseen items either . |
| Approach: | They propose a method for aligning simulated students with instructed ability to predict difficulty of open-ended items. |
| Outcome: | The proposed method outperforms existing methods on two real-world student responses. |
Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | Developing a theory of crowdsourcing for practical language problems remains an open challenge . |
| Approach: | This tutorial exposes NLP researchers to data collection crowdsourcing methods and principles through case studies. |
| Outcome: | This tutorial exposes NLP researchers to various data collection crowdsourcing methods and practices through case studies. |