Correcting Language Model Outputs by Editing Salient Layers (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models that use manual layer selection require prior domain knowledge and expensive empirical layer selection methods. |
| Approach: | They propose a model editing approach that selectively edits a small subset of model parameters to update the factual knowledge. |
| Outcome: | The proposed solution matches the accuracy of previous approaches with only 1/3 of their edits, enabling efficient updates to the parametric knowledge in large language models. |
Similar Papers
Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art methods for updating large language models require computational overhead and lack theoretical validation. |
| Approach: | They propose a model editing method that uses sentence concatenation with augmented random facts for generation regularization. |
| Outcome: | The proposed method outperforms state-of-the-art methods while maintaining generation quality and reducing computational overhead. |
Enhancing Semantic Consistency of Large Language Models through Model Editing: An Interpretability-Oriented Approach (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models generate inconsistent and sometimes contradictory outputs when presented with a prompt that has equivalent semantics but is expressed differently from the original prompt. |
| Approach: | They propose to refine a Large Language Model (LLM) with prompt-output pairs with equivalent semantics to achieve semantic consistency. |
| Outcome: | The proposed method improves the semantic consistency and task performance of LLMs. |
Knowledge Editing for Large Language Models (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are not immune to issues of factual accuracy or logically consistent. |
| Approach: | This tutorial will present cutting-edge methods and practical tools for editing Large Language Models (LLMs). |
| Outcome: | The aim of this course is to familiarize researchers with the latest advancements and emerging strategies in the realm of knowledge editing for LLMs. |
Editing Large Language Models: Problems, Methods, and Opportunities (2023.emnlp-main)
Copied to clipboard
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, Ningyu Zhang
| Challenge: | Recent advances in model editing for LLMs have created challenges and opportunities for the community. |
| Approach: | They propose to alter the behavior of LLMs efficiently within a specific domain without negatively impacting performance across other inputs. |
| Outcome: | The proposed method alters behavior of LLMs efficiently within a specific domain without negatively impacting performance across other inputs. |
Robust and Scalable Model Editing for Large Language Models (2024.lrec-main)
Copied to clipboard
Yingfa Chen, Zhengyan Zhang, Xu Han, Chaojun Xiao, Zhiyuan Liu, Chen Chen, Kuai Li, Tao Yang, Maosong Sun
| Challenge: | Existing methods that ignore contextual knowledge fail to reliably fall back to parametric knowledge when presented with irrelevant context. |
| Approach: | They propose to use contextual knowledge to update and correct LLMs' knowledge by in-context editing instead of retraining. |
| Outcome: | The proposed method outperforms current state-of-the-art methods by a large margin on a dataset that contains irrelevant questions. |
DUnE: Dataset for Unified Editing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models are susceptible to errors necessitating a comprehensive retraining process. |
| Approach: | They propose to define an edit as any natural language expression that solicits a change in the model’s outputs. |
| Outcome: | The proposed editing benchmarks show that retrieval-augmented language modeling outperforms specialized editing techniques and neither set of approaches has fully solved the generalized editing problem covered by the proposed benchmark. |
DocMEdit: Towards Document-Level Model Editing (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models only output short phrases or sentences, raising doubts about their practical usability. |
| Approach: | They propose a dataset focused on document-level model editing that aims to correct errors and outdated knowledge in Large language models (LLMs) they propose to use document-based model editing to improve model capabilities in real-world scenarios. |
| Outcome: | The proposed model editing task improves model capabilities in real-world scenarios and reduces the cost of retraining. |
Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling (D18-1)
Copied to clipboard
| Challenge: | Existing efforts to train pre-trained language models have brought significant improvements to various NLP applications. |
| Approach: | They propose to compress bulky LMs while preserving useful information for a specific task. |
| Outcome: | The proposed method can detach any layer without affecting others, and stretch shallow and wide LMs to be deep and narrow. |
Neuron-Level Sequential Editing for Large Language Models (2025.acl-long)
Copied to clipboard
Houcheng Jiang, Junfeng Fang, Tianyu Zhang, Baolong Bi, An Zhang, Ruipeng Wang, Tao Liang, Xiang Wang
| Challenge: | Existing model editing methods focus on single-round editing and often face significant challenges in sequential model editing. |
| Approach: | They propose a model editing method that optimizes the target layer’s hidden states using the model’s original weights to prevent model failure. |
| Outcome: | The proposed method outperforms existing model editing methods and is available on the open-source platform 4open.science. |
SSS: Editing Factual Knowledge in Language Models towards Semantic Sparse Space (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to modify LMs suffer from sub-optimal locality, where irrelevant neighborhood examples can be adversely influenced. |
| Approach: | They propose to use a model editing method to modify specific examples in LMs to improve locality and reasoning capability by directing the hidden state of edit example towards spaces where semantics are sparse. |
| Outcome: | The proposed method improves locality and reasoning capability on two datasets. |