Continual Machine Reading Comprehension via Uncertainty-aware Fixed Memory and Adversarial Domain Adaptation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing stationary-trained MRC systems are usually trained with in-domain data but are applied to new domain data. |
| Approach: | They propose a continual machine reading comprehension model with uncertainty-aware fixed memory and adversarial domain adaptation that keeps a stable understanding by learning both memory and new domain data. |
| Outcome: | The proposed model is superior to strong baselines and has a substantial incremental learning ability without catastrophically forgetting under two different continual MRC settings. |
Similar Papers
Adversarial Domain Adaptation for Machine Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | Existing models for machine reading comprehension rely on large amounts of human-annotated in-domain data. |
| Approach: | They propose an unsupervised domain adaptation framework for Machine Reading Comprehension where the source domain has a large amount of labeled data, while only unlabeled passages are available in the target domain. |
| Outcome: | The proposed framework can be generalizable to different MRC models and datasets and can be extended to semi-supervised learning. |
InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have identified the severe performance decrease on analogous classes as a key factor for catastrophic forgetting. |
| Approach: | They propose a replay-based continual text classification method that uses fast-slow and current-past contrastive learning to perform mutual information maximization and better recover previously learned representations. |
| Outcome: | The proposed method achieves state-of-the-art on three text classification tasks. |
Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions (2022.emnlp-main)
Copied to clipboard
| Challenge: | Currently, continuous learning methods suffer from catastrophic forgetting problem, causing model to forget previous knowledge while learning new knowledge. |
| Approach: | They propose a two-stage continuous learning method based on local features of the real loss to avoid catastrophic forgetting problem. |
| Outcome: | The proposed method achieves significant improvements on domain adaptation and more challenging language adaptation tasks. |
Advancing SMoE for Continuous Domain Adaptation of MLLMs: Adaptive Router and Domain-Specific Loss (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have explored Continual Instruction Tuning (CIT) in Multimodal Large Language Models (MLLMs), with a primary focus on Task-incremental CIT, where MLLM are required to continuously acquire new tasks. |
| Approach: | They propose a Sparse Mixture of Expert (SMoE) based method for domain-incremental CIT in Multimodal Large Language Models (MLLMs) . they equip the SMoA module with a domain-specific autoregressive loss (DSAL) they establish a new benchmark to evaluate the efficacy of their method . |
| Outcome: | The proposed method outperforms all baselines and is based on a Sparse Mixture of Experts (SMoE) module . |
ICL: Iterative Continual Learning for Multi-domain Neural Machine Translation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on learning domain knowledge from multiple domains, but task-specific parameters hinder mutual transfer of knowledge between new domains. |
| Approach: | They propose an iterative Continual learning framework for multi-domain neural machine translation that leverages previously acquired domain knowledge. |
| Outcome: | The proposed model outperforms baseline models on UM-Corpus and OPUS datasets. |
DuReader_robust: A Chinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applications (2021.acl-short)
Copied to clipboard
| Challenge: | In order to comprehensively verify the robustness and generalization of MRC models, we construct a real-world Chinese dataset - DuReader_robust . |
| Approach: | They introduce a real-world Chinese dataset to evaluate the robustness and generalization of MRC models from three aspects: over-sensitivity, over-stability and generalisation. |
| Outcome: | The proposed model fails to perform well on the challenge test set and may provide suggestions for future model development. |
Continual Lifelong Learning in Natural Language Processing: A Survey (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to continual learning (CL) are costly and time-consuming. |
| Approach: | They propose to examine the problem of continual learning in NLP through the lens of various NLP tasks and provide a critical review of existing methods. |
| Outcome: | The proposed methods are critical to the development of CL models and provide a critical review of existing methods and datasets. |
Robust Uncertainty Quantification for Self-Evolving Large Language Models via Continual Domain Pretraining (2026.findings-acl)
Copied to clipboard
| Challenge: | Conformal prediction (CP) has shown promise in offering correctness guarantees for LLMs, but it faces major challenges in continual domain pretraining (CDP). |
| Approach: | They propose an adaptive rejection and non-exchangeable CP framework that allows the LLM to selectively abstain from answering when its confidence or competence shifts significantly. |
| Outcome: | Experiments show that the proposed framework improves performance under continuous domain pretraining scenarios. |
Benchmarking Robustness of Machine Reading Comprehension Models (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks only evaluate models' robustness under test-time perturbations or adversarial attacks. |
| Approach: | They propose a model-agnostic benchmark to evaluate models' robustness under adversarial attacks. |
| Outcome: | The proposed model-agnostic benchmark evaluates models under four different types of adversarial attacks. |
FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning (2026.acl-long)
Copied to clipboard
Yujie Feng, Hao Wang, Jian Li, Xu Chu, Zhaolu Kang, Yiran Liu, Yasha Wang, Philip S. Yu, Xiao-Ming Wu
| Challenge: | Continual learning (CL) for large language models (LLMs) aims to enable sequential knowledge acquisition without catastrophic forgetting. |
| Approach: | They propose a framework that aligns replay schedules with a model-centric notion of time. |
| Outcome: | Experiments on three benchmarks show that FOREVER consistently mitigates catastrophic forgetting. |