Domain Generalization via Switch Knowledge Distillation for Robust Review Representation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing models for review representations of unseen or anonymous users are limited by their in-domain nature. |
| Approach: | They propose to use in-domain user and product information to generalize reviews . they use switch knowledge distillation to learn review representations for unseen users . |
| Outcome: | The proposed model performs well for existing or anonymous unseen users. |
Similar Papers
Distilling Discrimination and Generalization Knowledge for Event Detection via Delta-Representation Learning (P19-1)
Copied to clipboard
| Challenge: | Current neural event detection approaches focus on trigger-centric representations, which work well on distilling discrimination knowledge, but poorly on learning generalization knowledge. |
| Approach: | They propose a Delta-learning approach to distill discrimination and generalization knowledge by incrementally learning and adaptively fusing event representation. |
| Outcome: | The proposed method significantly outperforms previous approaches on unseen/sparse trigger words and achieves state-of-the-art performance on ACE2005 and KBP2017 datasets. |
Domain Generalization via Causal Adjustment for Cross-Domain Sentiment Analysis (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to domain adaptation fail to generalize well on unknown test data. |
| Approach: | They propose a backdoor adjustment-based causal model to disentangle domain-specific and domain-invariant representations that play essential roles in tackling domain shift. |
| Outcome: | The proposed model disentangles domain-specific and domain-invariant representations that play essential roles in tackling domain shift. |
HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizability (2021.acl-long)
Copied to clipboard
| Challenge: | Using data augmentation to fine-tune pre-trained models with task-specific data has been shown to be ineffective and redundant during fine-timing. |
| Approach: | They propose a data augmentation technique to regularize pre-trained models and encourage them to learn more generalizable features by dropping contiguous spans during training. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark and consistently exhibits superior generalization performances on out-of distribution and challenging counterexamples. |
Not to Overfit or Underfit the Source Domains? An Empirical Study of Domain Generalization in Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to limit overfitting of training domains are rooted in this problem . domain generalization (DG) seeks to train models on a small number of source domains . |
| Approach: | They propose to use knowledge distillation to train models on a small number of source domains to maximize their zero-shot out-of-domain utility. |
| Outcome: | The proposed model learns its source domains better and has better out-of-domain generalization . the proposed model outperforms existing approaches that aim to limit overfitting . |
SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) has emerged as a popular method for compressing large language models due to high inference costs and memory requirements. |
| Approach: | They propose a method that integrates the teacher model during the student's sequence generation to reduce misguidance from the teacher. |
| Outcome: | Experiments on three model families and five instruction-following datasets show that SWITCH surpasses traditional methods, especially in the generation of long sequential data. |
Domain Generalization for Text Classification with Memory-Based Supervised Contrastive Learning (2022.coling-1)
Copied to clipboard
| Challenge: | Existing approaches to cross-domain text classification focus on one-to-one domain adaptation. |
| Approach: | They propose a framework for domain generalization that uses contrastive learning with a memory-saving queue. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on Amazon review sentiment datasets and rumour detection datasets. |
Natural Language Generation for Effective Knowledge Distillation (D19-61)
Copied to clipboard
| Challenge: | Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks. |
| Approach: | They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques . |
| Outcome: | The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability. |
Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in neural machine translation (NMT) research have found NMT models sensitive to distribution shift and adversarial examples. |
| Approach: | They propose a leave-one-domain-out training strategy that learns to combine domain-specific parameters to avoid information leaking. |
| Outcome: | The proposed method outperforms baselines on three language pairs on average. |
Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Overconfidence in model generalization and calibration has been shown to impair model generalisation and calibration. |
| Approach: | They propose a regularization scheme that takes model probability into account and takes it into account . they use a prior label distribution to smooth target labels . |
| Outcome: | The proposed model improves model generalization and calibration by taking model probability into account. |
How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge Distillation (KD) is a model compression algorithm that helps transfer knowledge in a large neural network into a smaller one. |
| Approach: | They propose a framework to assess adversarial robustness of multiple KD algorithms. |
| Outcome: | The proposed algorithm achieves state-of-the-art on the GLUE benchmark and out-of domain generalization and adversarial robustness compared to competitive methods. |