Papers with sub-models
Recycling a Pre-trained BERT Encoder for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | In monolingual tasks, the number of unlearned model parameters is as huge as the number learned parameters in the BERT model. |
| Approach: | They propose to apply a pre-trained Bidirectional Encoder Representations from Transformers (BERT) model to Transformer-based neural machine translation (NMT) based on the Transformer. |
| Outcome: | The proposed model is stable and efficient in low-resource settings. |
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to decompose complex tasks into simpler ones do not require annotated decompositions. |
| Approach: | They propose a framework for building interpretable systems that learn to solve complex tasks by decomposing existing models into simpler ones solvable by existing models. |
| Outcome: | The proposed framework is more versatile than existing explainable systems for DROP and HotpotQA datasets, is more robust than state-of-the-art blackbox (uninterpretable) systems, and generates more understandable and trustworthy explanations compared to prior work. |
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large language models excel at understanding and generating human-like text, but their widespread deployment can be prohibitively expensive. |
| Approach: | They propose a method that makes large language models dynamic without Pre-Training . they use modularity in networks and sort sub-models based on computation/accuracy in a nested manner. |
| Outcome: | The proposed method can make large language models dynamic without pre-training and replace standard fine-tuning with sorted fine- tuning. |
Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models may not follow human instructions and produce toxic, hallucinated, or biased content. |
| Approach: | They propose a disperse-then-merge framework that dispersers instruction-following data into portions and trains multiple sub-models using different data portions. |
| Outcome: | The proposed framework outperforms data curation and training regularization on standard knowledge and reasoning benchmarks. |
ReasonRec: A Reasoning-Augmented Multimodal Agent for Unified Recommendation (2026.findings-acl)
Copied to clipboard
Yihua Zhang, Mingfu Liang, Jiyan Yang, Rong Jin, Wen-Yen Chen, Yiping Han, Huayu Li, Buyun Zhang, Liang Luo, Luke Simon, Sijia Liu, Tianlong Chen, Xi Liu
| Challenge: | Recent advances in multimodal recommenders lack explicit reasoning and self-awareness of uncertainty. |
| Approach: | They propose a reasoning-augmented multimodal agent structured around a three-stage explicit reasoning pipeline. |
| Outcome: | The proposed agent improves ranking metrics and performance on four standard recommendation tasks across five real-world datasets. |
R-AT: Regularized Adversarial Training for Natural Language Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, adversarial training is a popular and powerful regularization method in the natural language domain. |
| Approach: | They propose to regularize adversarial training via dropout by perturbing word embeddings . they find that R-AT can improve many models by reducing adversariality . |
| Outcome: | The proposed method can reduce the inconsistency between training and testing of models with dropout. |
Scaling Vision-Language Models with Sparse Mixture of Experts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study explores the effectiveness of mixture-of-experts (MoE) techniques in scaling vision-language models . alayrac and colleagues demonstrate the effectiveness and performance of MoE in scaling VLMs . |
| Approach: | They propose to use sparsely-gated mixture-of-experts techniques to scale vision-language models . they show that MoE can achieve state-of the-art performance over dense models a range of benchmarks . |
| Outcome: | The proposed approach achieves state-of-the-art performance over dense models of equivalent computational cost. |
Towards Robustifying NLI Models Against Lexical Dataset Biases (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies show that deep learning models exploit dataset biases without deep understanding of the language semantics. |
| Approach: | They propose two methods to debiase models against lexical dataset biases . they use contradiction-word bias and word-overlapping bias as examples . |
| Outcome: | The proposed method removes label bias at embedding level, while the other uses a bag-of-words sub-model to capture features likely to exploit the bias. |
Collaboration of Fusion and Independence: Hypercomplex-driven Robust Multi-Modal Knowledge Graph Completion (2026.acl-long)
Copied to clipboard
| Challenge: | Existing multi-modal knowledge graphs lack modality-specific information and are limited in their ability to capture nuanced semantic interplay between modalities. |
| Approach: | They propose a multi-modal knowledge graph completion method which integrates both paradigms . they use a fine-grained Entity Representation Factorization module and a Robust Relation-aware Modality Fusion module to obtain robust representations for three independent modalities and one fused modality. |
| Outcome: | The proposed method achieves coexistence and collaboration of fused and independent modality representations while maintaining modality-specific information. |