Dual Fixed-Size Ordinally Forgetting Encoding (FOFE) for Competitive Neural Language Models (D18-1)
Copied to clipboard
| Challenge: | In this paper, we propose a new approach to employ the fixed-size ordinally-forgetting encoding (FOFE) in neural languages modelling, called dual-FOFE. |
| Approach: | They propose a new approach to employ the fixed-size ordinally-forgetting encoding (FOFE) in neural languages modelling, called dual-FOFE. |
| Outcome: | The proposed method significantly reduces the complexity and improves perplexity by 10% over the original FOFE model. |
Similar Papers
BLUR: A Bi-Level Optimization Approach for LLM Unlearning (2026.eacl-long)
Copied to clipboard
Hadi Reisizadeh, Jinghan Jia, Zhiqi Bu, Bhanukiran Vinzamuri, Anil Ramakrishna, Kai-Wei Chang, Volkan Cevher, Sijia Liu, Mingyi Hong
| Challenge: | Existing algorithms to unlearn knowledge and capabilities from large datasets are unclear how to best formulate the unlearning problem. |
| Approach: | They propose to model the hierarchical structure of the unlearning problem, where the forget problem takes priority over the retain problem, and propose an algorithm that aims to unlearn knowledge and capabilities. |
| Outcome: | The proposed algorithm outperforms all state-of-the-art algorithms across unlearning tasks, models, and metrics. |
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging (2025.emnlp-main)
Copied to clipboard
Bowen Wang, Haiyuan Wan, Liwen Shi, Chen Yang, Peng He, Yue Ma, Haochen Han, Wenhao Li, Tiao Tan, Yongjian Li, Fangming Liu, Gong Yifan, Sheng Zhang
| Challenge: | Existing models that require task labels or performance trade-offs are susceptible to catastrophic forgetting. |
| Approach: | They propose a representation-aware model merging framework for continual learning without access to historical data. |
| Outcome: | The proposed framework outperforms baselines in knowledge retention and generalization across five NLP tasks and multiple continual learning scenarios. |
Code-switched Language Models Using Dual RNNs and Same-Source Pretraining (D18-1)
Copied to clipboard
| Challenge: | Using recurrent neural networks to build language models for code-switched text is an important problem with implications to downstream applications such as speech recognition and machine translation. |
| Approach: | They propose a novel recurrent neural network unit with dual components that focus on each language in the code-switched text separately and a generative model estimated using the training data. |
| Outcome: | The proposed techniques yield significant reductions in perplexity on Mandarin-English task and improve on baseline models. |
Encoding and Decoding Language in the Brain with Language Models (2026.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial introduces brain-language model alignment and recent advances in brain-informed fine-tuning and brain-based fine-caching with language models. |
| Approach: | This tutorial introduces brain-language model alignment and recent advances in brain-informed fine-tuning and scaling with language models. |
| Outcome: | This tutorial introduces brain-language model alignment and recent advances in brain-informed fine-tuning and decoding with language models. |
How to represent a word and predict it, too: Improving tied architectures for language modelling (D18-1)
Copied to clipboard
| Challenge: | Recent state-of-the-art models use word embeddings as input and output mappings instead of tied models. |
| Approach: | They propose to decouple hidden state from word embedding prediction . they extend their proposed modification to word2vec models . |
| Outcome: | The proposed architectures achieve comparable or better results compared to previous models without tying . the proposed architecture reduces parameters, enabling more compact models and faster learning. |
Modeling LLM Unlearning as an Asymmetric Two-Task Learning Problem (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are inherently dual-use and can be leveraged for both beneficial and harmful purposes. |
| Approach: | They propose a retention-prioritized gradient synthesis framework that decouples task-specific gradient extraction from conflict-aware combination. |
| Outcome: | The proposed method achieves tighter alignment on WMDP Bio and RWKU benchmarks. |
Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal instruction tuning often causes catastrophic forgetting of the base LLM’s language ability, even in strong models like Llama3. |
| Approach: | They propose a training-free parameter fusion framework that locates important parameters and selectively merges them. |
| Outcome: | The proposed framework preserves the influence of neurons with large parameter shifts while attenuating the influence . of neurons likely responsible for newly acquired visual capabilities while mitigating language degradation. |
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (N19-1)
Copied to clipboard
| Challenge: | Existing transfer learning methods employ language models pretrained on large generic corpora, but results come at a high computational cost and require task-specific architectures. |
| Approach: | They propose a transfer learning approach that combine a task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. |
| Outcome: | The proposed method surpasses well established transfer learning methods with greater level of complexity on a variety of affective and text classification tasks surpassing well established methods with higher level of difficulty. |
Mitigating Catastrophic Forgetting in Language Transfer via Model Merging (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have shown remarkable capabilities, particularly in English, but for less prevalent languages, performance can be significantly lower, making additional adaptation paramount. |
| Approach: | They propose a new adaptation method based on iteratively merging multiple models fine-tuned on a subset of available training data that reduces forgetting while maintaining learning on the target domain. |
| Outcome: | The proposed method outperforms LLAMA-3-8B-based models in German and German while maintaining learning on the target domain. |
Decoding with Limited Teacher Supervision Requires Understanding When to Trust the Teacher (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated their tremendous capability to generate human-like text sentences that convey rich knowledge in various problem domains. |
| Approach: | They propose an algorithm to aggregate small-scale LLM and LLM predictions on initial tokens so that the generated tokens can more accurately condition the subsequent token generation by small-level LLM only. |
| Outcome: | The proposed method improves on the limited supervision scenario on a wide range of models and datasets. |