Calibrating Imbalanced Classifiers with Focal Loss: An Empirical Study (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Imbalanced data distributions can cause models to overfit to majority classes and output unreliable (mostly overconfident) predictions. |
| Approach: | They propose to streamline the model development and deployment using focal loss to address imbalanced data distributions. |
| Outcome: | The proposed model training with focal loss improves calibration and accuracy compared to standard cross-entropy loss. |
Similar Papers
A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing (2023.eacl-main)
Copied to clipboard
| Challenge: | Developing methods to improve model performance in imbalanced data settings has been an active area for decades . |
| Approach: | They propose to use sampling, data augmentation, choice of loss function, staged learning, or model design to address class imbalance in NLP. |
| Outcome: | The proposed approaches are evaluated on a variety of NLP tasks or in the computer vision community. |
Focal Training and Tagger Decouple for Grammatical Error Correction (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for Grammatical Error Correction (GEC) are categorized into sequence-to-sequence approaches, tagging-based approaches, and hybrid approaches. |
| Approach: | They propose to decouple error detection layer from label tagging layer and to down-weight label imbalance and tabbing entanglement loss using Focal Loss. |
| Outcome: | The proposed methods are effective over three latest Chinese Grammatical Error Correction datasets. |
On the Importance of Adaptive Data Collection for Extremely Imbalanced Pairwise Tasks (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent datasets heuristically choose examples to ensure label balance . state-of-the-art models trained on QQP and WikiQA have only 2.4% average precision . |
| Approach: | They show that recent datasets heuristically choose examples to ensure label balance . they instead use active learning to retrieve uncertain points from a large pool of unlabeled utterance pairs . |
| Outcome: | The proposed model improves on QQP and WikiQA by using more informative negative examples. |
Outlier-Aware Training for Improving Group Accuracy Disparities (2022.aacl-srw)
Copied to clipboard
| Challenge: | Methods addressing spurious correlations such as Just Train Twice involve reweighting a subset of the training set to maximize the worst-group accuracy. |
| Approach: | They propose to reweight a subset of a training set to maximize the worst-group accuracy by detecting outliers and removing them before reweighing. |
| Outcome: | The proposed method achieves competitive or better accuracy compared with JTT and can detect and remove annotation errors in the subset being reweighted in JTT. |
Fairness-aware Class Imbalanced Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on class imbalance and mitigating bias have focused on the latter . a skewed class distribution hurts the performance of deep learning models, and is often referred to as "stereotyping" |
| Approach: | They propose to extend a margin-loss based approach to enforce fairness by using tweet sentiment and occupation classification to mitigate class imbalance and demographic bias. |
| Outcome: | The proposed methods help mitigate class imbalance and demographic biases through controlled experiments. |
On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work shows that deep learning models are sensitive to low-level correlations between simple features and specific output labels, leading to over-fitting and lack of generalization. |
| Approach: | They propose to eliminate single-word correlations altogether to mitigate this problem . they highlight several alternatives to dataset balancing to enhance contexts . |
| Outcome: | The proposed approach to balancing datasets is insufficient, the authors argue . they suggest enhancing datasets with richer contexts and abstaining from interaction . |
Learning on Imbalanced Noisy Data via Debiased Sample Selection and LLM-Driven Annotation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to learning with noisy labels are prone to selection bias and training bias . obtaining large-scale high-quality datasets is expensive and time-consuming in practical scenarios . |
| Approach: | They propose an imbalanced learning with noisy labels task to let model learn from noisy labels . they first conduct debiased sample selection to better separate clean samples from noisy samples . then they feed selected clean samples to active annotator large language models for re-annotating noisy samples. |
| Outcome: | The proposed method is superior to existing methods on synthetic and real-world datasets. |
Text Classification Under Class Distribution Shift: A Survey (2026.eacl-long)
Copied to clipboard
| Challenge: | ML models assume that training and test data are sampled from the same distribution, but in daily practice, this assumption is often broken. |
| Approach: | They survey articles studying open-set text classification to understand the distribution shifts and mitigation approaches for each problem setup. |
| Outcome: | The proposed methods can solve problems caused by the shifting class distribution in open-set text classification and related tasks. |
A Study on the Calibration of In-context Learning (2024.naacl-long)
Copied to clipboard
Hanlin Zhang, YiFan Zhang, Yaodong Yu, Dhruv Madeka, Dean Foster, Eric Xing, Himabindu Lakkaraju, Sham Kakade
| Challenge: | Prior research has demonstrated improvements in the calibration of language models (LMs) in-context learning is a popular method for adapting static LMs to safety-critical domains. |
| Approach: | They use in-context learning to adapt static language models through tailored prompts to a wide range of tasks and find that miscalibration occurs in low-shot settings. |
| Outcome: | The proposed calibrations show that models exhibit increased miscalibration before achieving better calibration in low-shot settings. |
Mitigating the Burden of Redundant Datasets via Batch-Wise Unique Samples and Frequency-Aware Losses (2023.acl-industry)
Copied to clipboard
Donato Crisostomi, Andrea Caciolai, Alessandro Pedrani, Kay Rottmann, Alessandro Manzotti, Enrico Palumbo, Davide Bernardi
| Challenge: | Existing solutions to train deep learning models on redundant datasets are difficult to implement in industrial settings. |
| Approach: | They propose a method to eliminate duplicates at the batch level without altering the data distribution observed by the model. |
| Outcome: | The proposed approach reduces training times on models on redundant datasets by up to 87% and 46% on average, with a drop in model performance of 0.2% relative at worst. |