Papers by Yihong Liu
Copied to clipboard
| Challenge: | Large Language Models excel at code generation by learning from vast code corpora, but a fundamental semantic gap remains between training on textual patterns and the goal of functional correctness . reinforcement learning with verifiable rewards (RLVR) approaches are inefficient for establishing a well-aligned connection between the textual representation of code and its execution semantics. |
| Approach: | They propose a novel approach that integrates execution semantics alignment into the RLVR training pipeline for code generation. |
| Outcome: | The proposed model outperforms baseline training and RLVR and shows strong applicability across RL and LLMs. |
Copied to clipboard
| Challenge: | Existing model-based channel prediction methods suffer from limited accuracy due to imperfect temporal modeling, while existing AI-based methods suffers from limited generalization due to inadequate training strategies. |
| Approach: | They propose a generative pre-trained language model for channel prediction based on channel correlation and train it based upon transformer decoder architecture. |
| Outcome: | The proposed model can learn various channel characteristics and perform impressive tasks across multiple dimensions. |
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) improves the fine-tuning efficiency and performance of large language models. |
| Approach: | They propose a low-rank adaptive approach that decomposes update matrix into global and local adapters and assigns them to local and global adapters. |
| Outcome: | The proposed method achieves up to 2.7% accuracy improvement over LoRA and its variants on commonsense reasoning, mathematical reasoning, and code generation. |
Copied to clipboard
| Challenge: | Existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of Large Language Models (LLMs). |
| Approach: | They propose a repository-level benchmark named DevEval to evaluate LLMs' coding abilities in real-world code repositories. |
| Outcome: | The proposed benchmarks show that the LLMs perform better in real-world code repositories than existing benchmarks. |
Copied to clipboard
| Challenge: | Multilingual language models store factual knowledge across languages but struggle to provide consistent responses to semantically equivalent prompts in different languages. |
| Approach: | They propose a linear shortcut method that bypasses computations in the final layers . this method improves accuracy and cross-lingual consistency . |
| Outcome: | The proposed method improves prediction accuracy and cross-lingual consistency. |
Copied to clipboard
| Challenge: | Large language models excel at generating English counterfactuals but their effectiveness in generating multilingual counterfacts remains unclear. |
| Approach: | They conduct automatic evaluations on both directly generated and derived counterfactuals in six languages and find that cross-lingual perturbations follow common strategic principles. |
| Outcome: | The proposed models show that translation-based counterfactuals offer higher validity than their directly generated counterparts, but still fall short of matching the quality of the original English counterf actuals. |
Copied to clipboard
| Challenge: | Recent work shows that large reasoning models arrive at the correct answer before completing textual reasoning steps, indicating the presence of latent reasoning. |
| Approach: | They conduct a systematic investigation of multilingual latent reasoning in large reasoning models across 11 languages. |
| Outcome: | The proposed model arrive at the correct answer before completing the reasoning steps, indicating the presence of latent reasoning. |
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly deployed in multilingual, real-world applications where user inputs introduce typographical errors. |
| Approach: | They propose a multilingual typo generation algorithm that simulates human-like errors based on language-specific keyboard layouts and typing behavior. |
| Outcome: | The proposed model can generate the correct answer ("500") under typos in English, German, and Russian. |
Copied to clipboard
| Challenge: | Several studies have focused on programming languages in a monolingual setting, but most focus on programming language models. |
| Approach: | They perform a few-shot translation task on 21 PL pairs using two Llama-based models and decode the embeddings of intermediate layers. |
| Outcome: | The proposed model assigns high probability to English tokens in the second half of the intermediate layers and language-specific neurons are concentrated in the bottom layers . the model's concept space is closer to English (including PL keywords) and the model is more efficient at identifying language-related neurons. |
Copied to clipboard
| Challenge: | retrieval-augmented generation (RAG) is a powerful tool for NLP applications . but it is challenging to encode large knowledge bases as compact offline structures . |
| Approach: | They propose a coarse-to-fine hierarchical graph inference method that uses random walks to retrieve information from a corpus of documents. |
| Outcome: | The proposed method reduces offline indexing costs and accelerates retrieval. |
Copied to clipboard
| Challenge: | Existing curriculum learning approaches rely on manually defined difficulty metrics which may not accurately reflect the model’s own perspective. |
| Approach: | They propose a self-adaptive curriculum learning paradigm that prioritizes fine-tuning examples based on difficulty scores predicted by pre-trained language models (PLMs) they evaluate four datasets covering binary and multi-class classification tasks. |
| Outcome: | The proposed model leads to faster convergence and improved performance compared to standard random sampling. |
Copied to clipboard
| Challenge: | Recent advances in deep learning have boosted the development of neural machine translation (NMT). |
| Approach: | They propose a flow-adapter architecture for unsupervised neural machine translation that leverages normalizing flows to model distributions of sentence-level latent representations. |
| Outcome: | The proposed model achieves competitive results on several unsupervised MT benchmarks. |
Copied to clipboard
| Challenge: | Existing studies focus on English-centric aspects of sentiment analysis, limiting scope for multilingual evaluation and research. |
| Approach: | They propose to use a multilingual dataset to analyze aspects with associated sentiment elements in text. |
| Outcome: | The proposed dataset is the most extensive multilingual parallel dataset for ABSA to date. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) demonstrate their utility in character simulations, but they pose a risk of generating unsafe content. |
| Approach: | They propose a method which dynamically adjusts safety-utility preferences based on the degree of risk coupling and guides the model to generate responses biased toward utility or safety. |
| Outcome: | The proposed method improves safety metrics while maintaining utility. |
Copied to clipboard
| Challenge: | Recent studies show that post-aligning multilingual pretrained language models improve crosslingual alignment, but it is unclear how and why this is achieved. |
| Approach: | They propose to explicitly evaluate crosslingual alignment by adding transliterations to models using original and transliterated data. |
| Outcome: | The proposed approach improves crosslingual alignment even for random sentences. |
Copied to clipboard
| Challenge: | Conceptualizer is a method that creates a bipartite directed alignment graph between source language concepts and sets of target language strings. |
| Approach: | They propose a method that creates a bipartite directed alignment graph between source language concepts and sets of target language strings. |
| Outcome: | The proposed method has good alignment accuracy across all languages and on 32 Swadesh concepts. |
Copied to clipboard
| Challenge: | Existing frameworks for benchmarking in NLP often overestimate performance . however, manually creating a variety of test types requires significant human labor . |
| Approach: | They propose a framework that leverages large language models to generate a wide range of test types . they first generate sentences via LLMs and then identifies challenging examples . |
| Outcome: | The proposed framework overestimates performance on two classification tasks. |
Copied to clipboard
| Challenge: | Existing methods to pretrain multilingual models are limited by the number of embedding parameters and the complexity of the model. |
| Approach: | They propose a framework that initializes the embeddings of unseen subwords and can adapt a model to multiple languages. |
| Outcome: | The proposed framework can adapt a pre-trained model to multiple languages efficiently and effectively. |
Copied to clipboard
| Challenge: | Existing benchmarks only evaluate LLMs' abilities for task completion as assistant AI. |
| Approach: | They propose a dialogue evaluation benchmark that contains 12 dialogue tasks to evaluate LLMs' capabilities as human-like dialogue systems. |
| Outcome: | The proposed benchmark contains 12 tasks to evaluate LLMs' capabilities . it shows that instruction tuning improves human likeness, but not as human-like systems . |
Copied to clipboard
| Challenge: | Existing mPLMs that handle non-transliterated data are not sufficient to train crosslingual models. |
| Approach: | They propose a framework to transliterate related languages into a common script by exploiting existing mPLMs and their tokenizer without any training. |
| Outcome: | The proposed framework can create strong baselines for data that is transliterated into a common script by exploiting an existing mPLM and its tokenizer without any training. |
Copied to clipboard
| Challenge: | TrickCatcher generates test cases that pass existing tests yet contain bugs . a recent study found that tricky bugs are not detected by test suites . |
| Approach: | They propose an LLM-powered approach to generating test cases for uncovering bugs in plausible programs . they use a PUT and specification to generate program variants, an input generator and an Llm to construct test inputs . |
| Outcome: | The proposed approach achieves recall, precision, and F1 scores that are 1.80, 2.65, and 1.66 . trickCatcher generates program variants based on the program under test and its specification . |
Copied to clipboard
| Challenge: | Recent work has examined final-answer accuracy in multilingual settings, but the behavior of thinking traces, i.e., the intermediate steps that lead to the final answer, remains underexplored. |
| Approach: | They propose to measure language compliance, answer accuracy, and answer consistency when LRMs are explicitly instructed or prompt-hacked to think in a target language. |
| Outcome: | The proposed model improves in English and other high-resource languages while relying on traces to varying degrees. |
Copied to clipboard
| Challenge: | In-context machine translation (MT) with large language models can take advantage of linguistic resources such as grammar books and dictionaries. |
| Approach: | They propose to use in-context machine translation (MT) with large language models to take advantage of linguistic resources such as grammar books and dictionaries. |
| Outcome: | The proposed approach can take advantage of dictionaries and grammar books, but its performance is poor for many lowresource languages. |
Copied to clipboard
| Challenge: | Existing chit-chat systems tend to generate uninformative responses and lack coherent personality traits due to the diversity of speakers. |
| Approach: | They propose a transmitter-receiver framework which explicitly models understanding between interlocutors. |
| Outcome: | The proposed framework improves on a large public dataset, Persona-Chat, with a significant boost over the state-of-the-art frameworks. |
Copied to clipboard
| Challenge: | Existing approaches to reinforcement learning with verifiable reward (RLVR) are limited by difficulty or lack of exploration. |
| Approach: | They propose a self-evolving curriculum learning framework based on chain-of-thought reasoning optimization that constrains exploration space by self-generating and verifying CoT trajectories. |
| Outcome: | The proposed framework enables LLMs to solve previously unsolved problems without external supervision and is compatible with various RL fine-tuning methods. |
Copied to clipboard
| Challenge: | Empathy improves human-machine dialogue systems by enhancing the user's experience. |
| Approach: | They propose a framework that leverages specialized encoders to capture the key features of emotion, cause, and commonsense and collaboratively models these through a Conditional Variational Auto-Encoder. |
| Outcome: | Empirical results show that the framework outperforms baseline models and offers a robust solution for empathetic dialogue generation. |
Copied to clipboard
| Challenge: | Current legalAI tasks divide sentencing and legal reasoning into two separate tasks, resulting in inconsistency between the reasoning and predictions. |
| Approach: | They propose a new task that generates both legal reasoning and sentencing decisions using a framework that applies structured legal chains to guide the model through comprehensive case assessments. |
| Outcome: | The proposed model outperforms baseline models on real-world, open-source Chinese legal case datasets. |
Copied to clipboard
| Challenge: | Recent studies have focused on content repetition, but structural repetition is a more prevalent problem in code generation. |
| Approach: | They propose a decoding approach that eliminates repetition problems in code generation by identifying grammar rules and strategically decaying the likelihood of critical tokens that contribute to repetitions. |
| Outcome: | The proposed approach outperforms baselines and humanEval benchmarks on CodeRepetEval dataset and MBPP benchmarks, effectively reducing repetitions and enhancing the quality of generated code. |
Copied to clipboard
| Challenge: | Existing approaches to combat character hallucination are vulnerable to attack . large language models (LLMs) are capable of generating responses inconsistent with intended personas . |
| Approach: | They propose a novel defence strategy that generates supplemental context through narration to mitigate role-query conflicts and improve query generalization. |
| Outcome: | The proposed defence strategy outperforms refusal-based strategies in character hallucinations and query generalization. |
Copied to clipboard
| Challenge: | The world’s more than 7000 languages are written in at least 293 scripts, which poses a difficulty for multilingual pretrained language models in learning crosslingual knowledge through lexical overlap. |
| Approach: | They propose a framework that optimizes the Transliteration Contrastive Modeling objective to fine-tune an mPLM by contrasting sentences in its training data and transliterations in a unified script. |
| Outcome: | The proposed model outperforms Glot500-m on zero-shot crosslingual transfer tasks while retaining uniformity across scripts. |
Copied to clipboard
| Challenge: | Combinatorial optimization has long been dominated by manually engineered heuristics, which require substantial expert intuition and implementation overhead. |
| Approach: | They propose a framework that couples an island migration model with elite selection to maintain population diversity. |
| Outcome: | The proposed framework achieves superior accuracy on the Traveling Salesman and Bin Packing Problems. |
Copied to clipboard
| Challenge: | In large language models, certain neurons can store distinct pieces of knowledge learned during pretraining. |
| Approach: | They hypothesize that relation-specific neurons detect relation in input text and guide generation involving such a relation. |
| Outcome: | The proposed model can handle facts involving relation r and facts containing a different relation . |
Copied to clipboard
| Challenge: | Pre-trained language models exhibit suboptimal performance on mid- and low-resource languages due to limited exposure to these languages during pre-training. |
| Approach: | They propose a similarity-based subword embedding initialization heuristic that introduces new tokens specific to target languages, initializes their embedders, and applies continual pre-training on target-language data. |
| Outcome: | The proposed method outperforms random initialization baseline and matches or exceeds OFA in both continual pre-training convergence and downstream task performance. |
Copied to clipboard
| Challenge: | Considering the vast size and wide-ranging sources of LLMs’ training data, it could explicitly or implicitly include test data. |
| Approach: | They propose a Contamination Detection via output Distribution (CDD) which detects data contamination only by identifying the peakedness of LLM's output distribution. |
| Outcome: | The proposed method improves performance by 21.8%-30.2% on humanEval and TED: trustworthy evaluation via output distribution. |
Copied to clipboard
| Challenge: | Recent multilingual pretrained language models often avoid using language embeddings, which places a significant burden on token representations to encode all language-specific information. |
| Approach: | They propose a method that incorporates both language and script embeddings into the output of Transformer blocks before passing the final representations to the language modeling head for prediction. |
| Outcome: | The proposed method outperforms the baseline model in zero-shot crosslingual transfer across diverse downstream tasks. |
Copied to clipboard
| Challenge: | Existing LegalAI tasks are descriptive or predictive, requiring the users to translate the information into legal reasoning. |
| Approach: | They propose a task to generate a structured defence opinion conditioned jointly on an indictment and the defendant’s stated opinion, which often present conflicting claims. |
| Outcome: | The proposed approach improves on eight large language models (LLMs) and shows that it is more efficient than previous approaches. |
Copied to clipboard
| Challenge: | Large Language Models are capable of recalling multilingual factual knowledge, but most studies evaluate only the final model, leaving the development of factual recall and crosslingual consistency unexplored. |
| Approach: | They trace how factual recall and crosslingual consistency evolve during pretraining, focusing on OLMo-7B as a case study. |
| Outcome: | The results show that fact frequency is the key to a better recall of multilingual facts, regardless of language, and some low-frequency facts in non-English languages can still be correctly recalled. |
Copied to clipboard
| Challenge: | Existing language modeling methods rely on large-scale text data to learn the sequential patterns of words. |
| Approach: | They propose to use sememes to represent the implicit semantics behind words for language modeling . they propose to employ sememe-driven language models to fine-grained semem-level semantics . |
| Outcome: | Experiments on language modeling and the downstream application of headline generation show the effectiveness of SDLM. |
Copied to clipboard
| Challenge: | High-quality, diverse data are vital for large language models (LLMs) but remain scarce and costly. |
| Approach: | They define the first HSS domain system covering 14 mainstream fields and introduce HSS-Synth. |
| Outcome: | the proposed pipeline outperforms 14 leading baselines on 16 benchmarks. |
Copied to clipboard
| Challenge: | Existing training methods for code generation do not improve code correctness and efficiency. |
| Approach: | They propose a framework that integrates preference learning into code generation to improve code correctness and efficiency. |
| Outcome: | The proposed framework improves code correctness and efficiency by integrating preference learning into code generation. |
Copied to clipboard
| Challenge: | Existing work on colexification patterns relies on annotated word lists, limiting scalability and usefulness in NLP. |
| Approach: | They propose two methods to train multilingual graphs from colexification patterns using an unannotated parallel corpus. |
| Outcome: | The proposed methods achieve high recall on CLICS and transfer learning in multilingual graphs. |
Copied to clipboard
| Challenge: | Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs). |
| Approach: | They propose a hybrid-policy optimization approach that synergizes internal exploitation with external data to achieve stronger reasoning capabilities. |
| Outcome: | The proposed approach achieves state-of-the-art performance on six math reasoning benchmarks and superior performance on out-of distribution reasoning tasks. |
Copied to clipboard
| Challenge: | Recent mPLMs have shown impressive performance on crosslingual transfer tasks . however, the performance is often hindered when a lowresource target language is written in a different script than the high-resource source language. |
| Approach: | They propose a transliteration-based method to improve cross-lingual alignment between languages using diverse scripts. |
| Outcome: | The proposed method outperforms the original model on Englishcentric transfer tasks up to 50%. |