Domain-Agnostic Neural Architecture for Class Incremental Continual Learning in Document Processing Platform (2023.acl-industry)
Copied to clipboard
| Challenge: | Recent methods with stochastic gradient learning struggle in streaming data setups and are restricted to specific domains. |
| Approach: | They propose a fully differentiable architecture that enables the training of high-performance classifiers when examples from each class are presented separately. |
| Outcome: | The proposed architecture achieves SOTA results without a memory buffer and clearly outperforms the reference methods. |
Similar Papers
Continual Learning with Semi-supervised Contrastive Distillation for Incremental Neural Machine Translation (2024.acl-long)
Copied to clipboard
| Challenge: | Multi-domain learning is a good solution for solving domain tasks but it requires retraining when adding a new domain. |
| Approach: | They propose to exploit unlabeled data from the same distributions of the older domains to avoid catastrophic forgetting. |
| Outcome: | The proposed framework exploits unlabeled data from the same distributions of the older domains to avoid catastrophic forgetting. |
Continuous Learning for Large-scale Personalized Domain Classification (N19-1)
Copied to clipboard
| Challenge: | Domain classification is the task to map spoken language utterances to one of the natural language understanding domains in intelligent personal digital assistants. |
| Approach: | They propose a neural-based approach for continuous domain adaption with normalization and regularization to accommodate new domains. |
| Outcome: | The proposed approach outperforms baseline methods on accommodated new domains and existing known domains by a large margin. |
Efficient Large-Scale Neural Domain Classification with Personalized Attention (P18-1)
Copied to clipboard
| Challenge: | Using a scalable neural model, we show that personalization improves domain classification accuracy in a setting with thousands of overlapping domains. |
| Approach: | They propose a scalable neural model architecture with a shared encoder that incorporates personalization information and domain-specific classifiers that solves the problem efficiently. |
| Outcome: | The proposed architecture achieves two orders of magnitude faster than full model retraining. |
Advancing SMoE for Continuous Domain Adaptation of MLLMs: Adaptive Router and Domain-Specific Loss (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have explored Continual Instruction Tuning (CIT) in Multimodal Large Language Models (MLLMs), with a primary focus on Task-incremental CIT, where MLLM are required to continuously acquire new tasks. |
| Approach: | They propose a Sparse Mixture of Expert (SMoE) based method for domain-incremental CIT in Multimodal Large Language Models (MLLMs) . they equip the SMoA module with a domain-specific autoregressive loss (DSAL) they establish a new benchmark to evaluate the efficacy of their method . |
| Outcome: | The proposed method outperforms all baselines and is based on a Sparse Mixture of Experts (SMoE) module . |
Transformer Based Multi-Source Domain Adaptation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve machine learning performance are mixed experts and domain adversarial training. |
| Approach: | They investigate the problem of unsupervised multi-source domain adaptation . they combine predictions of multiple domain experts and combine them to induce a domain agnostic representation space . |
| Outcome: | The proposed methods improve models' performance while limiting learning time. |
Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing continual learning approaches suffer from low accuracy and performance fluctuation when the distributions of old and new data are significantly different. |
| Approach: | They propose a hyperparameter-free continual learning model for text data that can stably produce high performance under various environments. |
| Outcome: | The proposed model outperforms the best state-of-the-art method by 20% in average accuracy and each component contributes effectively to overall performance. |
Compact Personalized Models for Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | a large proportion of model parameters can be frozen during adaptation with minimal or no reduction in translation quality. |
| Approach: | They propose gradient-based domain adaptation methods for self-attentive machine translation models . they encourage structured sparsity in the set of offset tensors during learning . |
| Outcome: | The proposed method achieves high space and time efficiency using sparse models . the results compare the proposed method with incremental adaptation . |
Domain Differential Adaptation for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural networks are data hungry and domain sensitive, so it is difficult to obtain labeled data for every domain. |
| Approach: | They propose a framework for domain adaptation where we model the difference between domains instead of smoothing over them. |
| Outcome: | The proposed framework improves on domain adaptation in multiple experimental settings. |
Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts (2024.findings-acl)
Copied to clipboard
Ganesh Jawahar, Haichuan Yang, Yunyang Xiong, Zechun Liu, Dilin Wang, Fei Sun, Meng Li, Aasish Pappu, Barlas Oguz, Muhammad Abdul-Mageed, Laks Lakshmanan, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Neural architecture search (NAS) uses weight-sharing supernets to generate diverse subnetworks without retraining. |
| Approach: | They propose a weight-sharing supernet that leverages mixture-of-experts to enhance supernet model expressiveness with minimal training overhead. |
| Outcome: | The proposed method achieves state-of-the-art (SoTA) performance in NAS for fast machine translation models, surpassing NAS-BERT and AutoDistil across various model sizes. |
DoMIX: An Efficient Framework for Exploiting Domain Knowledge in Fine-Tuning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for domain-adaptive pre-training (DAP) face several limitations: high computational cost and GPU memory usage during training; and lack of generalized model for all end tasks. |
| Approach: | They propose a domain-adaptive pre-training (DAP) method that uses a representative parameter-efficient fine-tuning method to provide pre-trained models for specific tasks. |
| Outcome: | The proposed method can be extended beyond the DAP setting to standard LLM fine-tuning scenarios. |