| Challenge: | Recent research has shown that black-box testing is not applicable to NLP models. |
| Approach: | They propose a set of white-box testing methods that are customized for transformer-based NLP models and adapt them to a black-box test suite. |
| Outcome: | The proposed methods can reduce testing suites by 60% while retaining failing tests, thereby concentrating faultdetection power of the test suite. |
Similar Papers
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020.acl-main)
Copied to clipboard
| Challenge: | In a recent study, we show that holding-out data can overestimate performance of NLP models. |
| Approach: | They propose a task-agnostic methodology for testing NLP models using a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation. |
| Outcome: | The proposed method identifies critical failures in commercial and state-of-the-art models. |
TestAug: A Framework for Augmenting Capability-based NLP Tests (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work on capability-based testing requires the developer to compose each individual test template from scratch. |
| Approach: | They propose a capability-based NLP testing framework that requires the developer to only annotate a few test templates while leveraging the GPT-3 engine to generate the majority of test cases. |
| Outcome: | The proposed framework saves the developer's manual efforts and guarantees the correctness of the generated suites with a validity checker. |
Adaptive Testing and Debugging of NLP Models (2022.acl-long)
Copied to clipboard
| Challenge: | Current approaches to testing and debugging NLP models rely on variable human creativity and extensive labor to instantiate bugs. |
| Approach: | They propose a process which uses large scale language models to automatically write unit tests highlighting bugs in a target model. |
| Outcome: | The proposed process makes users 5-10x more effective at finding bugs than current approaches. |
Whispers of Doubt Amidst Echoes of Triumph in NLP Robustness (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to measure robustness are problematic, and out-of-domain evaluations are no longer relevant. |
| Approach: | They examine models of different sizes spanning different architectural choices and pretraining objectives. |
| Outcome: | The results show that not all out-of-domain tests provide insight into robustness . merely scaling models does not make them adequately robust . |
SynthEval: Hybrid Behavioral Testing of NLP Models with Synthetic Evaluation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks for benchmarking in NLP often overestimate performance . however, manually creating a variety of test types requires significant human labor . |
| Approach: | They propose a framework that leverages large language models to generate a wide range of test types . they first generate sentences via LLMs and then identifies challenging examples . |
| Outcome: | The proposed framework overestimates performance on two classification tasks. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |
How to Dissect a Muppet: The Structure of Transformer Embedding Spaces (2022.tacl-1)
Copied to clipboard
| Challenge: | Pretrained embeddings based on the Transformer architecture have taken the NLP community by storm . a novel decomposition of Transformer output embeddables is demonstrated . |
| Approach: | They propose to decompose Transformer output embeddings into a sum of vector factors . they show multi-head attentions and feed-forwards are not equally useful in downstream applications . |
| Outcome: | The proposed method outperforms recurrent architectures on a wide variety of tasks. |
RealFormer: Transformer Likes Residual Attention (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing techniques to create Residual Attention Layer Transformer networks outperform the canonical Transformer on a wide spectrum of tasks. |
| Approach: | They propose a technique to create Residual Attention Layer Transformer networks that outperform the canonical Transformer on a wide spectrum of tasks. |
| Outcome: | The proposed technique outperforms the canonical Transformer on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQUAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP. |
Can Explanations Be Useful for Calibrating Black Box Models? (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models are often used as black boxes to adapt to new domains, but there is no single recipe for making them work. |
| Approach: | They propose to use black box models to improve their performance on new domains by leveraging explanations of their behavior. |
| Outcome: | The proposed method improves model generalization performance on two tasks using explanations. |
Fixing Model Bugs with Natural Language Patches (2022.emnlp-main)
Copied to clipboard
| Challenge: | a growing body of research focused on using language to give instructions, supervision and even inductive biases to models instead of relying exclusively on labeled examples. |
| Approach: | They explore natural language patches that provide corrective feedback at the right level of abstraction. |
| Outcome: | The proposed model improves accuracy on real data by 1–4 accuracy points on different slices of a sentiment analysis dataset and F1 by 7 points on a relation extraction dataset. |