Papers by Yekun Chai
CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models (LLMs) are limited by their narrow language pairs and tasks, failing to adequately assess their code-mixing abilities. |
| Approach: | They propose a benchmark to assess large language models' (LLMs) code-mixing abilities that covers eight tasks and 18 languages from seven language families. |
| Outcome: | The proposed method combines word substitution with GPT-4 prompting to generate large-scale synthetic code-mixed texts. |
Debiasing Multilingual LLMs in Cross-lingual Latent Space (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have evaluated their cross-lingual transferability by directly applying these methods to LLM representations, revealing their limited effectiveness across languages. |
| Approach: | They propose to perform debiasing in a joint latent space rather than directly on LLM representations by using an autoencoder trained on parallel TED talk scripts. |
| Outcome: | The proposed method improves both the overall debiasing performance and cross-lingual transferability of the proposed techniques across four languages. |
Predicate-Argument Based Bi-Encoder for Paraphrase Identification (2022.acl-long)
Copied to clipboard
| Challenge: | Cross-encoders have been widely applied to sentence pair tasks . bi-encodings have significantly lower computation complexity and are better suited to symmetric tasks. |
| Approach: | They propose a bi-encoder approach to the paraphrase identification task and incorporate predicate-argument information into SBERT. |
| Outcome: | The proposed model outperforms SBERT/SRoBERTa on six paraphrase identification datasets with a minimal increase in parameters. |
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code (2025.coling-industry)
Copied to clipboard
Taishi Nakamura, Mayank Mishra, Simone Tedeschi, Yekun Chai, Jason T. Stillerman, Felix Friedrich, Prateek Yadav, Tanmay Laud, Vu Minh Chien, Terry Yue Zhuo, Diganta Misra, Ben Bogin, Xuan-Son Vu, Marzena Karpinska, Arnav Varma Dantuluri, Wojciech Kusa, Tommaso Furlanello, Rio Yokota, Niklas Muennighoff, Suhas Pai, Tosin Adewumi, Veronika Laippala, Xiaozhe Yao, Adalberto Barbosa Junior, Aleksandr Drozd, Jordan Clive, Kshitij Gupta, Liangyu Chen, Qi Sun, Ken Tsui, Nour Moustafa-Fahmy, Nicolo Monti, Tai Dang, Ziyang Luo, Tien-Tung Bui, Roberto Navigli, Virendra Mehta, Matthew Blumberg, Victor May, Hiep Nguyen, Sampo Pyysalo
| Challenge: | Pretrained language models are integral part of AI applications, but their high computational cost limits accessibility. |
| Approach: | They evaluate Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code. |
| Outcome: | The proposed model outperforms existing models on English, Finnish, Hindi, Japanese, Vietnamese, and code. |
Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work does not take full advantage of over-parameterized characteristics of large pre-trained language models. |
| Approach: | They propose a method that uses frozen "thinned" networks to obtain a mixture of rewards and advance the derivative-free prompt learning. |
| Outcome: | The proposed method outperforms previous gradient-free prompt learning methods and achieves parity with gradient-based counterparts on seven language understanding benchmarks under few-shot settings. |
Autoregressive Pre-Training on Pixels and Texts (2024.emnlp-main)
Copied to clipboard
| Challenge: | pixel-based language modeling integrates visual and textual data to improve performance of language models. |
| Approach: | They propose a method that integrates visual and textual data into an autoregressive framework. |
| Outcome: | The proposed method improves performance of pixel-based language models by incorporating visual and textual data. |
Tokenization Falling Short: On Subword Robustness in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary. |
| Approach: | They propose to tokenize raw text into sequences of subword identifiers from a predefined vocabulary . they also investigate the challenges and their impact on large language models . |
| Outcome: | The proposed model can mitigate tokenization issues, but still suffer from typos and other variations. |
Understanding Subword Compositionality of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) take sequences of subwords as input, requiring them to compose subword representations into meaningful word-level representations. |
| Approach: | They propose to probe how large language models compose subword information . they find structural similarity, semantic decomposability, and form retention are key aspects . |
| Outcome: | The proposed models can be classified into three distinct groups, the authors show . they show that they can achieve great performance when probing layer by layer their sensitivity to semantic decompositionality . |
Highway Transformer: Self-Gating Enhanced Self-Attentive Networks (2020.acl-main)
Copied to clipboard
| Challenge: | Self-attention mechanisms have made striking state-of-the-art (SOTA) progress in various sequence learning tasks, attending to all the global contexts at different locations. |
| Approach: | They propose a gated component self-dependency units (SDU) that incorporates LSTM-styled gating units to replenish internal semantic importance within the multi-dimensional latent space of individual representations. |
| Outcome: | The proposed system could boost the performance of the Transformer modules by allowing them to skip connections and step towards suboptimal points during the optimization process. |
On Training Data Influence of GPT Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | generative language models have redefined performance standards across tasks . current research on the influence of training data on autoregressivity remains underexplored . |
| Approach: | They propose a parameterized simulation to assess the impact of training examples on the training dynamics of GPT models. |
| Outcome: | The proposed approach compares existing methods with existing methods across training scenarios in generative language models, spanning tasks across 14 million to 2.8 billion parameters. |
EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing key-value (KV) cache compression methods ignore interplays between layer-specific feature patterns and task performance. |
| Approach: | They propose an adaptive framework for layer-wise, task-driven KV cache compression that optimizes memory efficiency and task performance. |
| Outcome: | EvolKV outperforms baseline methods on long-context tasks and surpasses heuristics by 7 percentage points on GSM8K. |
Graph-Augmented Open-Domain Multi-Document Summarization (2025.coling-industry)
Copied to clipboard
| Challenge: | Existing methods for summarizing documents neglect the relationships between documents . existing methods treat retrieval and summarization as separate tasks . |
| Approach: | They propose a framework that captures global document relationships through graph-based clustering . this cluster-level thematic information is then used to guide large language models . |
| Outcome: | The proposed framework significantly improves retrieval accuracy and produces better summaries than existing methods. |
Counter-Contrastive Learning for Language GANs (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Generative Adversarial Networks (GANs) have proven to be difficult to generate natural language due to the uninformative learning signals passed from the discriminator. |
| Approach: | They propose to adopt the counter-contrastive learning method to support the generator’s training in language GANs by pulling the language representations of generated and real samples together and pushing apart representations. |
| Outcome: | The proposed method outperforms existing GANs on synthetic and real benchmarks and yields competitive performance compared to previous methods. |
ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | ERNIE-Code is a unified pre-trained language model for 116 NLs and 6 PLs. |
| Approach: | They propose a unified pre-trained language model for 116 NLs and 6 PLs . they employ span-corruption language modeling that learns patterns from monolingual NL or PL . |
| Outcome: | The proposed model outperforms previous multilingual models for NL or NL across end tasks. |
HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing benchmarks focus on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). |
| Approach: | They propose a benchmark to evaluate multilingual LLMs using multiple natural languages. |
| Outcome: | The proposed benchmarks focus on translating English prompts to multilingual code or have been constrained to very limited natural languages (NLs). |
Curiosity-Driven Reinforcement Learning from Human Feedback (2025.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models with human preferences, but often at the cost of reduced output diversity. |
| Approach: | They propose a framework that incorporates intrinsic rewards for novel states alongside traditional sparse extrinsic rewards to optimize both output diversity and alignment quality. |
| Outcome: | The proposed framework achieves significant gains in diversity on multiple diversity-oriented metrics while maintaining alignment with human preferences comparable to standard RLHF. |