Is There a One-Model-Fits-All Approach to Information Extraction? Revisiting Task Definition Biases (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Definition bias is a negative phenomenon that can mislead models. |
| Approach: | They propose a framework that measures definition bias, bias-aware fine-tuning and task-specific bias mitigation to mitigate definition bias in information extraction. |
| Outcome: | The proposed framework mitigates definition bias in information extraction tasks by measuring definition bias, bias-aware fine-tuning, and task-specific bias mitigation. |
Similar Papers
Beyond Performance: Quantifying and Mitigating Label Bias in LLMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit undesirable preference toward predicting certain answers over others, despite their adaptability to diverse tasks. |
| Approach: | They propose a label bias calibration method that outperforms recent calibration approaches for improving performance and mitigating label bias. |
| Outcome: | The proposed method outperforms calibration approaches for improving performance and mitigating label bias. |
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2020.acl-main)
Copied to clipboard
| Challenge: | a growing number of studies address the effect of bias on predictions, but no unifying framework exists . a general phenomenon of biased predictive models in NLP is not recent, authors say . |
| Approach: | They propose a unifying framework for identifying and reducing bias in natural language processing . they propose to differentiate two consequences of bias and four potential origins of bias . |
| Outcome: | The proposed framework provides an overview of predictive bias in natural language processing . it differentiates two consequences of bias and four potential origins of bias: label bias, selection bias, model overamplification, and semantic bias. |
Cognitive Effects and Biases in Large Language Models (2026.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial bridges psychology and NLP to clarify cognitive effects and biases in large language models. |
| Approach: | This tutorial bridges psychology and NLP to clarify cognitive effects and biases in large language models. |
| Outcome: | This tutorial bridges psychology and NLP to clarify cognitive effects and biases in large language models. |
When and Why Does Bias Mitigation Work? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural models exploit shallow surface features to perform language understanding tasks, rather than learning the deeper language understanding and reasoning skills that practitioners desire. |
| Approach: | They propose to use model debiasing techniques to pressure models away from spurious features and to use them to learn useful representations instead. |
| Outcome: | The proposed methods increase models' reliance on hidden biases instead of learning robust features that help them solve a task. |
Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitions (2025.emnlp-main)
Copied to clipboard
Seyedali Mohammadi, Bhaskara Hanuma Vedula, Hemank Lamba, Edward Raff, Ponnurangam Kumaraguru, Francis Ferraro, Manas Gaur
| Challenge: | Exact label definitions are considered as clues to disambiguate unclear labels, helping models perform their tasks more effectively. |
| Approach: | They conducted controlled experiments on multiple explanation benchmark datasets and label definition conditions using expert-curated, LLM-generated, perturbed, and swapped definitions. |
| Outcome: | The results suggest that models often default to internal representations, particularly in general tasks, while domain-specific tasks benefit more from explicit definitions. |
LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs (2025.naacl-long)
Copied to clipboard
Do Xuan Long, Ngoc-Hai Nguyen, Tiviatis Sim, Hieu Dao, Shafiq Joty, Kenji Kawaguchi, Nancy F. Chen, Min-Yen Kan
| Challenge: | Using format-following capabilities, state-of-the-art large language models (LLMs) can be leveraged to tailor outputs to specific task formats. |
| Approach: | They propose to define a format bias evaluation metric and establish effective strategies to reduce it. |
| Outcome: | The proposed evaluation reduces the variance in ChatGPT’s performance among wrapping formats from 235.33 to 0.71 (%2) |
A Closer Look at Data Bias in Neural Extractive Summarization Models (D19-54)
Copied to clipboard
| Challenge: | In this paper, we examine the generalization behaviour of summarization models . we propose several properties of datasets that matter for generalization . |
| Approach: | They propose several properties of datasets which matter for generalization of summarization models. |
| Outcome: | The proposed approach improves the state-of-the-art model by rethinking the model design process on a typical dataset. |
Automatic Error Analysis for Document-level Information Extraction (2022.acl-long)
Copied to clipboard
| Challenge: | Document-level information extraction (IE) tasks have been revisited in earnest . evaluation of the approaches has been limited in a number of dimensions . |
| Approach: | They propose a transformation-based framework for automating error analysis in document-level event and (N-ary) relation extraction. |
| Outcome: | The proposed framework compares two state-of-the-art document-level template-filling approaches on datasets from three domains and four systems from the MUC-4 evaluation. |
On Event Individuation for Document-Level Information Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a bomb exploded in a restaurant in Lima, and a second device was deactivated by the police . |
| Approach: | They argue that the task demands definitive answers to thorny questions of *event individuation* they argue that even human experts disagree on the task . |
| Outcome: | The proposed task demands definitive answers to thorny questions of *event individuation* . the proposed task also raises concerns about the usefulness of template filling metrics . |
Mitigating Label Biases for In-context Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to categorize label biases in in-context learning (ICL) have not addressed all three types of label bias. |
| Approach: | They propose a method that estimates a language model’s label bias using random in-domain words from the task corpus to categorize and detect label biases in ICL. |
| Outcome: | The proposed method significantly improves the performance of GPT-J and GPT-3 on a wide range of tasks. |