Papers with MoTE
Mixture of insighTful Experts (MoTE): The Synergy of Reasoning Chains and Expert Mixtures in Self-Alignment (2025.acl-long)
Copied to clipboard
Zhili Liu, Yunhao Gou, Kai Chen, Lanqing Hong, Jiahui Gao, Fei Mi, Yu Zhang, Zhenguo Li, Xin Jiang, Qun Liu, James Kwok
| Challenge: | Recent studies show that reasoning abilities contribute significantly to model safety, while integrating Mixture-of-Experts (MoE) architectures can further enhance alignment. |
| Approach: | They propose a framework that synergistically combines reasoning chains and expert mixtures to improve self-alignment. |
| Outcome: | The proposed framework improves model safety, jailbreak resistance, and over-refusal capabilities, achieving performance comparable to OpenAI’s state-of-the-art o1 model. |
Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Experts (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing classification models only consider the temporal variations of existing data . current models focus on English corpora, leaving time as domains unexplored . |
| Approach: | They propose a framework to generalize classifiers over time on four languages, English, Danish, French, and German. |
| Outcome: | The proposed framework can generalize classifiers over time on four languages, English, Danish, French, and German. |
Beyond instruction-conditioning, MoTE: Mixture of Task Experts for Multi-task Embedding Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Instruction-conditioning has become the dominant approach for embedding specialization, but its direct application to low-capacity models imposes representational constraints that limit the performance gains derived from specialization. |
| Approach: | They propose a mixture of task experts transformer block which leverages task-specialized parameters trained with Task-Aware Contrastive Learning to enhance the model’s ability to generate specialized embeddings. |
| Outcome: | The proposed model achieves 64% higher performance gains in retrieval datasets (+3.27 +5.21) and 43% higher performance gain across all datasets (+1.81 2.60). |