Qianlan Ying, Payal Bajaj, Budhaditya Deb, Yu Yang, Wei Wang, Bojia Lin, Milad Shokouhi, Xia Song, Yang Yang, Daxin Jiang
| Challenge: | We consider scaling automated suggested replies (SR) to multiple languages for a commercial email application. |
| Approach: | They propose a multi-lingual multi-task continual learning framework with auxiliary tasks and language adapters to train universal language representation across regions. |
| Outcome: | The proposed model reduces catastrophic forgetting and improves cross-lingual transfer across languages while reducing training costs. |
Similar Papers
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in natural language processing (NLP) have led to significant breakthroughs in the field. |
| Approach: | They evaluate ChatGPT over multiple tasks with diverse languages and large datasets to provide more comprehensive information for multilingual NLP applications. |
| Outcome: | The proposed model can process and generate texts for multiple languages due to its multilingual training data. |
A Survey of Post-Training Scaling in Large Language Models (2025.acl-long)
Copied to clipboard
Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, Jie Tang
| Challenge: | Large language models (LLMs) have demonstrated proficiency in understanding and generating human natural languages. |
| Approach: | They propose a framework for scaling large language models using supervised fine-tuning, RLxF and test-time compute methodologies. |
| Outcome: | The proposed model can be used to understand and generate human natural languages. |
Scaling Laws for Multilingual Language Models (2025.findings-acl)
Copied to clipboard
Yifei He, Alon Benhaim, Barun Patra, Praneetha Vaddamanu, Sanchit Ahuja, Parul Chopra, Vishrav Chaudhary, Han Zhao, Xia Song
| Challenge: | Existing scaling laws for language models are limited to a limited number of languages, but they can be applied to arbitrary number of different languages. |
| Approach: | They propose a scaling law for general-purpose decoder-only language models trained on multilingual data that shifts focus from individual languages to language families. |
| Outcome: | The proposed scaling law can be applied to models trained on multilingual data . it can be used to predict performance across multiple languages and models . |
Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems (D19-56)
Copied to clipboard
| Challenge: | Statistical conversational systems are complex, timeintensive, expensive, and not easily transferable due to data scarcity. |
| Approach: | They propose a task-oriented dialogue model that operates on text input . they validate it on multi-domain task-orientated dialogues from a multi-word dataset . |
| Outcome: | The proposed model bypasses explicit policy and language generation modules on multi-domain task-oriented dialogues from the MultiWOZ dataset. |
SCALE: Upscaled Continual Learning of Large Language Models (2026.findings-acl)
Copied to clipboard
Jin-woo Lee, Junhwa Choi, Bongkyu Hwang, Jinho Choo, Bogun Kim, Jeongseon Yi, Joonseok Lee, DongYoung Jung, Jaeseon Park, Kyoungwon Park, Suk-hoon Jung
| Challenge: | Recent discussions suggest that further progress will come from scaling the right structure, not merely parameters or data, while preserving acquired knowledge. |
| Approach: | They propose a width upscaling architecture that inserts lightweight expansions into linear modules while freezing all pre-trained parameters. |
| Outcome: | The proposed architecture reduces severe forgetting while learning new knowledge on a controlled synthetic biography benchmark. |
Towards Autonomous Tool Utilization in Language Models: A Unified, Efficient and Scalable Framework (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in tool learning for large language models have led to a new trend to allow LLMs to leverage external tools. |
| Approach: | They propose a framework for fine-tuning language models that categorizes queries into three different types . they also introduce an "instruct, execute, and reformat" strategy specifically designed for efficient data annotation . |
| Outcome: | The proposed framework surpasses open-source language models and GPT-3.5/4 on multiple evaluation metrics. |
Unifying Cross-Lingual Transfer across Scenarios of Resource Scarcity (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to deal with resource scarcity have not been developed to deal effectively with the problem. |
| Approach: | They propose to use a set of tools to harness data from one or more high-resource "source" languages to compensate for a shortage of data in low-resourced "target" languages. |
| Outcome: | The proposed technique can be easily adapted to unseen languages, extending the range of the proposed technique and translation-based transfer more broadly. |
Thesis Proposal: Targeted and Unified Cross-Lingual Unlearning from Multilingual Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | Large language models trained on corpora scraped from the web can reproduce sensitive and copyright-protected data. |
| Approach: | They propose to extend existing benchmarks to multilingual data by compiling parallel translations of question-answer pairs consisting of real-world facts and synthetic personally identifiable information. |
| Outcome: | The proposed dataset will include translations of question-answer pairs consisting of real-world facts and synthetic personally identifiable information. |
SCALE: Synergized Collaboration of Asymmetric Language Translation Engines (2024.findings-acl)
Copied to clipboard
| Challenge: | In this paper, we introduce SCALE, a collaborative framework that connects a compact Specialized Translation Model (STM) and a general-purpose Large Language Model (LLM) as one unified translation engine. |
| Approach: | They propose a collaborative framework that connects a Specialized Translation Model (STM) and a general-purpose Large Language Model (LLM) as one unified translation engine. |
| Outcome: | The proposed framework outperforms both LLMs and supervised models in high-resource or challenging low-resourced settings. |
Paraphrastic Representations at Scale (2022.emnlp-demos)
Copied to clipboard
| Challenge: | a new system allows users to train their own state-of-the-art paraphrastic sentence representations in a variety of languages. |
| Approach: | They propose a system that allows users to train their own paraphrastic sentence representations in a variety of languages. |
| Outcome: | The proposed models outperform previous models on monolingual and cross-lingual tasks and can be used on CPUs with little difference in inference speed. |