When and Why a Model Fails? A Human-in-the-loop Error Detection Framework for Sentiment Analysis (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing methods for sentiment analysis are difficult to assess for erroneous predictions that might exist prior to deployment. |
| Approach: | They propose a framework for error detection based on explainable features that can detect erroneous model predictions on unseen data with high precision. |
| Outcome: | The proposed framework detects erroneous model predictions on unseen data with high precision, given limited human-in-the-loop intervention, and can be deployed on unselected data with a high accuracy. |
Similar Papers
Financial Sentiment Analysis: An Investigation into Common Mistakes and Silver Bullets (2020.coling-main)
Copied to clipboard
| Challenge: | Recent dominance of machine learning-based natural language processing methods has overemphasized model accuracies rather than studying the reasons behind their errors. |
| Approach: | They investigate the error patterns of some widely acknowledged sentiment analysis methods in the finance domain. |
| Outcome: | The proposed models are based on the existing models and have important clues for improving them. |
Aspect-based Sentiment Analysis as Machine Reading Comprehension (2022.coling-1)
Copied to clipboard
| Challenge: | Existing approaches to aspect-based sentiment analysis stack multiple modules and result in severe error propagation. |
| Approach: | They propose a MRC-PrOmpt mOdeL framework where multiple sentiment aspects are elicited by a machine reading comprehension model and their corresponding sentiment polarities are classified in a prompt learning way. |
| Outcome: | The proposed framework significantly outperforms existing state-of-the-art models or achieves comparable performance on widely-used benchmark datasets. |
FIND: Human-in-the-Loop Debugging Deep Text Classifiers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models are limited in the number of available datasets and lack the necessary tools to improve them. |
| Approach: | They propose a framework which enables humans to debug deep learning text classifiers by disabling irrelevant hidden features. |
| Outcome: | Experiments show that using FIND, humans can improve CNN text classifiers trained on different types of imperfect datasets. |
Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments (2023.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations of human-in-the-loop systems to combat misinformation are often set up automatically using datasets that were retrospectively constructed. |
| Approach: | They propose a human-in-the-loop evaluation framework for fact-checking novel misinformation claims and identifying social media messages that support them. |
| Outcome: | The proposed framework is based on modern NLP methods for human-in-the-loop fact-checking in the domain of COVID-19 treatments. |
Which Spurious Correlations Impact Reasoning in NLI Models? A Visual Interactive Diagnosis through Data-Constrained Counterfactuals (2023.acl-demo)
Copied to clipboard
| Challenge: | a spurious correlation exists when a feature correlates with the target label while there is no causal relationship between the feature and the label. |
| Approach: | They propose a dashboard that allows users to generate diverse and challenging examples by drawing inspiration from GPT-3 suggestions. |
| Outcome: | The proposed dashboard enables users to generate diverse and challenging examples by drawing inspiration from GPT-3 suggestions and make refinements based on the feedback. |
Detecting Label Errors by Using Pre-Trained Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for label error detection focus on label errors in training data. |
| Approach: | They propose a method for introducing realistic, human-originated label noise into existing crowdsourced datasets such as SNLI and TweetNLP. |
| Outcome: | The proposed method outperforms existing methods for detecting label errors in natural language datasets. |
Automated Evaluation of Out-of-Context Errors (L18-1)
Copied to clipboard
| Challenge: | Existing methods to modify text understanding systems use only one sentence at a time . however, considering a larger context can improve performance for text understanding tasks. |
| Approach: | They propose to modify existing text data to insert out-of-context errors . they use a 2016 TEDTalk corpus to evaluate computational models for text understanding . |
| Outcome: | The proposed method targets real-world problems of transcription and translation systems by inserting authentic out-of-context errors. |
Improving In-Context Learning with Prediction Feedback for Sentiment Analysis (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved promising results in sentiment analysis through the in-context learning paradigm. |
| Approach: | They propose a framework that incorporates prior predictions and feedback to improve sentiment understanding by incorporating prior feedback and leveraging a feedback-driven prompt. |
| Outcome: | The proposed framework improves on nine sentiment analysis datasets with an average improvement of 5.95% over conventional methods. |
Analyzing Modality Robustness in Multimodal Sentiment Analysis (2022.naacl-main)
Copied to clipboard
| Challenge: | despite its importance, little attention has been paid to improving the robustness of multimodal models. |
| Approach: | They propose simple diagnostic checks for modality robustness in a trained multimodal model . they find MSA models highly sensitive to a single modality, which creates issues . |
| Outcome: | The proposed checks show that models are highly sensitive to a single modality, which creates issues in their robustness. |
How Reliable are Model Diagnostics? (2021.findings-acl)
Copied to clipboard
| Challenge: | Contemporary statistical models trade off interpretability and simplicity for powerful parameterizations and inductive biases, enabling impressive performance. |
| Approach: | They examine three recent models and find they are not yet reliable . they also formulate recommendations for practitioners and researchers . |
| Outcome: | The proposed models are not as reliable as previously assumed, the authors argue . their findings suggest that they are needed for improving models and training setups . |