| Challenge: | Existing methods that extract features from input text to explain a classifier's prediction are limiting to models that are faithful to their predictions. |
| Approach: | They propose a framework for guiding model explanations by supervising them explicitly using task-related lexicons to direct supervise model explanation. |
| Outcome: | The proposed method improves model explanations without sacrificing performance on sentiment analysis and toxicity detection tasks while demoting spurious correlations with African American English dialects. |
Similar Papers
Natural Language Inference with Mixed Effects (2020.starsem-1)
Copied to clipboard
| Challenge: | aggregating raw annotations to a single label is problematic due to disagreement among annotators. |
| Approach: | They propose a generic method that allows one to skip the aggregation step and train on the raw annotations directly without subjecting the model to unwanted noise. |
| Outcome: | The proposed method improves performance over models that do not incorporate such effects. |
Language models are not naysayers: an analysis of language models on negation benchmarks (2023.starsem-1)
Copied to clipboard
| Challenge: | Negation has been shown to be a major bottleneck for masked language models, such as BERT, but whether this finding still holds for larger-sized auto-regressive language models has not been studied comprehensively. |
| Approach: | They evaluate the ability of current-generation auto-regressive language models to handle negation using a wide range of benchmarks and models. |
| Outcome: | The proposed models are compared against a wide range of negation benchmarks and show that they are insensitive to negation, inability to capture the lexical semantics of negations, and failure to reason under negation. |
How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing studies on the performance of pre-trained language models on natural language understanding tasks have focused on the natural language inference and textual entailment tasks. |
| Approach: | They propose to use corrupted data to fine-tune pre-trained language models to assess their language understanding capabilities. |
| Outcome: | The proposed transformations can be applied to all but one NLU task and show that understanding the meaning of utterances is not required for high performance. |
Syntax and Semantics Meet in the “Middle”: Probing the Syntax-Semantics Interface of LMs Through Agentivity (2023.starsem-1)
Copied to clipboard
| Challenge: | a recent study examined how large language models handle interactions in meaning across words and larger syntactic forms. |
| Approach: | They propose to use a dataset to examine the linguistic properties of optionally transitive English verbs to examine their agentivity. |
| Outcome: | The proposed model outperforms all other models in the evaluation dataset . the results are better correlated with human judgements than syntactic and semantic corpus statistics . |
A Trip Towards Fairness: Bias and De-Biasing in Large Language Models (2024.starsem-1)
Copied to clipboard
| Challenge: | a little or a large bias in CtB-LLMs may cause huge harm . LLaMA and OPT families have an important bias in gender, race, religion, and profession. |
| Approach: | They propose to debiase three families of Very Large-Language Models with LORA to reduce bias by 4.12 points in the normalized stereotype score. |
| Outcome: | The proposed model reduces bias up to 4.12 points in the normalized stereotype score. |
Lexical Substitution as Causal Language Modeling (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods for lexical substitution task lacks autoregressive decoding capabilities. |
| Approach: | They propose a framework that uses causal language modeling (CLM) for lexical substitution task. |
| Outcome: | The proposed system outperforms GeneSis, the best previously published supervised LST method. |
Fine-tuning BERT with Focus Words for Explanation Regeneration (2020.starsem-1)
Copied to clipboard
| Challenge: | Existing approaches to explain the correct answer in multiple-choice QA are low in F-scores and lack of performance. |
| Approach: | They introduce a lightweight focus feature in a transformer-based NLP model and examine performance improvements. |
| Outcome: | The proposed model achieves the highest scores, second only to a computationally intensive system. |
Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge (2021.starsem-1)
Copied to clipboard
Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, Philippe Blache
| Challenge: | Prior work has explored the ability of computational models to predict word semantic fit with a given predicate. |
| Approach: | They compare Transformers Language Models to SDM to assess their performance . they found that TLMs do not capture important aspects of event knowledge . people can discriminate between typical and atypical events, they say . |
| Outcome: | The proposed models can achieve comparable performance to SDM, but they lack important aspects of event knowledge. |
A Generative Approach for Mitigating Structural Biases in Natural Language Inference (2022.starsem-1)
Copied to clipboard
| Challenge: | Natural language inference datasets contain artifacts and biases that allow models to perform poorly by using a biased subset of the input without considering the remainder features. |
| Approach: | They reformulate a natural language inference task as a generative task . they find that this approach is highly robust to large amounts of bias . |
| Outcome: | The proposed model is highly robust to large amounts of bias. |
Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures (2023.starsem-1)
Copied to clipboard
| Challenge: | a recent attempt at language modeling with predicted semantic structure failed to establish empirical lower bounds on what could have made the attempt successful. |
| Approach: | They propose a concise binary vector representation of semantic structure at the lexical level and evaluate how good an incremental tagger needs to be to achieve better-than-baseline performance. |
| Outcome: | The proposed model can achieve better-than-baseline performance without losing its main advantages and lower bounds on prediction quality can't be established via a single score alone. |