| Challenge: | Existing methods to modify text understanding systems use only one sentence at a time . however, considering a larger context can improve performance for text understanding tasks. |
| Approach: | They propose to modify existing text data to insert out-of-context errors . they use a 2016 TEDTalk corpus to evaluate computational models for text understanding . |
| Outcome: | The proposed method targets real-world problems of transcription and translation systems by inserting authentic out-of-context errors. |
Similar Papers
Automatic Error Analysis for Document-level Information Extraction (2022.acl-long)
Copied to clipboard
| Challenge: | Document-level information extraction (IE) tasks have been revisited in earnest . evaluation of the approaches has been limited in a number of dimensions . |
| Approach: | They propose a transformation-based framework for automating error analysis in document-level event and (N-ary) relation extraction. |
| Outcome: | The proposed framework compares two state-of-the-art document-level template-filling approaches on datasets from three domains and four systems from the MUC-4 evaluation. |
Correcting the Autocorrect: Context-Aware Typographical Error Correction via Training Data Augmentation (2020.lrec-1)
Copied to clipboard
| Challenge: | a recent study shows that typographical errors are now ubiquitous . traditional spelling correction software is inadequate to correct typographical mistakes . |
| Approach: | They propose to generate typographical errors based on annotated spelling errors . they then use annotations to introduce errors into substantially larger corpora . |
| Outcome: | The proposed method generates typographical errors that require context-aware error detection . it also shows that machine learning can correct typographical mistakes based on the data . |
Out-of-Context Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lightweight technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
| Approach: | They propose a lightweight technique that trains only new token embeddings on axioms . they train only new embeddables and evaluate them on unseen tasks . |
| Outcome: | The proposed technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
Editing Large Language Models: Problems, Methods, and Opportunities (2023.emnlp-main)
Copied to clipboard
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, Ningyu Zhang
| Challenge: | Recent advances in model editing for LLMs have created challenges and opportunities for the community. |
| Approach: | They propose to alter the behavior of LLMs efficiently within a specific domain without negatively impacting performance across other inputs. |
| Outcome: | The proposed method alters behavior of LLMs efficiently within a specific domain without negatively impacting performance across other inputs. |
Wronging a Right: Generating Better Errors to Improve Grammatical Error Detection (D18-1)
Copied to clipboard
| Challenge: | grammatical error correction is a labor-intensive task that requires large amounts of training data. |
| Approach: | They propose to use a human-annotated corpus of human-generated grammatical errors to generate a synthetic model. |
| Outcome: | The proposed method outperforms the current state of the art in grammatical error correction . human annotators achieve 39.39 F1 scores, suggesting the model generates mostly human-like instances . |
Automatic Evaluation of Attribution by Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Generative large language models (LLMs) incorporate external references to generate and support claims. however, evaluating the attribution remains an open problem. |
| Approach: | They investigate automatic evaluation of attribution given by large language models . they define different types of attributed errors and then explore two approaches . |
| Outcome: | The proposed methods highlight promising signals and challenges. |
The Model Agreed, But Didn’t Learn: Diagnosing Surface Compliance in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models internalize vast world knowledge as parametric memory, yet inherit the staleness and errors of their source corpora. |
| Approach: | They propose a framework that subjects models to discriminative self-assessment under diverse contextual pressures to scrutinize subtle behavioral nuances induced by memory modifications. |
| Outcome: | The proposed framework achieves high benchmarks without overwriting internal beliefs, while recursive modifications accumulate representational residues, triggering cognitive instability and permanently diminishing the reversibility of the model’s memory state. |
Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent model editing techniques can introduce large unwanted side effects, a new study shows . existing specificity benchmarks do not detect these unwanted side-effects . a recent study shows that model edits can cause significant performance drop . |
| Approach: | They extend existing CounterFact benchmark to include a dynamic component and propose a new benchmark to evaluate model editing techniques. |
| Outcome: | The proposed benchmark improves existing benchmarks for specificity and avoids unwanted side effects. |
Grammatical error detection in transcriptions of spoken English (2020.coling-main)
Copied to clipboard
| Challenge: | CrowdED corpus of spoken English monologues on business topics was crowdsourced from native speakers of English and learners of English with German as their first language. |
| Approach: | They propose to use the corpus recordings to correct existing speech transcriptions and edit them to make them more fluent. |
| Outcome: | The proposed transcription corrections and annotations can be used for automatic transcription post-editing and grammatical error correction for spoken English. |
Deconstructing In-Context Learning: Understanding Prompts via Corruption (2024.lrec-main)
Copied to clipboard
| Challenge: | Prior work examined how modifying different elements of the prompt can affect model performance, but this limited number of elements made replication challenging. |
| Approach: | They decompose the entire prompt into four components: task description, demonstration inputs, labels, and inline instructions provided for each demonstration. |
| Outcome: | The proposed model is robust to minor prompt modifications, but its underlying pre-trained backbone is brittle . previous studies focused on models with fewer than 15 billion parameters or exclusively examined black-box models like GPT-3 or PaLM, making replication challenging. |