| Challenge: | Modern NLP systems rely on offline training and are inefficient for new tasks. |
| Approach: | They propose a visually grounded ContinuaL learning task which simulates the continual acquisition of compositional phrases from streaming visual scenes. |
| Outcome: | The proposed system improves on existing systems, but it's infeasible to store all possible compositions. |
Similar Papers
Visually Grounded Continual Language Learning with Selective Specialization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on the role of model components in learning specialized or generalizable representations are lacking. |
| Approach: | They propose to analyze selection strategies for visually grounded continual language learning using two diagnostic datasets. |
| Outcome: | The proposed models outperform existing models and provide enough control and flexibility for a thorough model analysis. |
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing language pretraining frameworks only take the language context as selfsupervision . current frameworks do not take grounding information from the external visual world . |
| Approach: | They propose a visually-supervised language model that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to related images. |
| Outcome: | The proposed model improves on multiple pure-language tasks. |
Non-compositional Expression Generation and its Continual Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that pre-trained language models are limited in their ability to generate non-compositional expressions. |
| Approach: | They propose a mask-infilling task to examine non-compositional expressions in English . they compare large pre-trained language models and continual learning methods . |
| Outcome: | The proposed task aims to investigate the ability of pre-trained language models to generate non-compositional expressions in English and their continual learning. |
Non-compositional Expression Generation Based on Curriculum Learning and Continual Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Non-compositional expressions are a classic ‘pain in the neck’ for NLP systems because of their non-composibility and limited data resources. |
| Approach: | They propose a dynamic curriculum learning framework which learns training examples from easy ones to harder ones but suffers from the forgetting problem. |
| Outcome: | The proposed framework improves on idiomatic expression generation and metaphor generation. |
Exploring Continual Learning for Code Generation Models (2023.acl-short)
Copied to clipboard
Prateek Yadav, Qing Sun, Hantian Ding, Xiaopeng Li, Dejiao Zhang, Ming Tan, Parminder Bhatia, Xiaofei Ma, Ramesh Nallapati, Murali Krishna Ramanathan, Mohit Bansal, Bing Xiang
| Challenge: | Large-scale code generation models such as Copilot and CodeT5 are expensive to train and re-train. |
| Approach: | They propose a benchmark for Continual Learning (CL) that covers a wide range of tasks with different input and output programming languages. |
| Outcome: | The proposed method improves on Prompt Pooling with Teacher Forcing, which suffers catastrophic forgetting due to stark distribution shifts in coding tasks. |
SLoRA: Balancing Plasticity and Forgetting in Large Language Models for Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved remarkable success across diverse tasks through large-scale pretraining. |
| Approach: | They propose a framework that filters noisy components from LoRA updates via subspace similarity with the base model. |
| Outcome: | The proposed framework improves accuracy by 12%, reduces forgetting by 29%, and filters out over 30% of LoRA parameters identified as noisy. |
Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing continual learning setups for embodied intelligence focus on executing low-level actions, neglecting the ability to learn high-level planning and multi-level knowledge. |
| Approach: | They propose a Hierarchical Embodied Continual Learning Setups (HEC) that divides the agent’s continual learning process into two layers: high-level instructions and low-level actions. |
| Outcome: | The proposed method reduces the forgetting of old tasks compared to other methods, while orthogonally training the remaining parts. |
Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior (2021.tacl-1)
Copied to clipboard
| Challenge: | Despite its potential and prevalence, this signal is understudied for learning to generate natural language. |
| Approach: | They propose to use this signal to improve the system's ability to generate instructions via contextual bandit learning. |
| Outcome: | The proposed system improves its ability to generate natural language through interaction with users, and the results are shown. |
Cross-lingual Continual Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing multi-lingual representations such as the one-hop transfer learning pipeline are difficult to adapt to new languages. |
| Approach: | They propose a cross-lingual continuum learning paradigm that evaluates continuous learning approaches that adapt to emerging data from different languages. |
| Outcome: | The proposed model can be used to adapt to new languages in a sequential manner. |
Data-Centric Continual Pre-training for 500+ Languages: A New Bilingual Translation Corpus and Multilingual Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models pre-trained on massive data have promoted multilingual natural language processing (NLP). |
| Approach: | They construct a bilingual translation corpus with 2,500 language pairs and develop a suite of four models with parallel data. |
| Outcome: | The proposed model suites are evaluated across 7 tasks and 12 benchmarks. |