Papers by Xiang Ye
Copied to clipboard
| Challenge: | Recent commercial systems such as Suno demonstrate strong capabilities in long-form song generation, but academic research remains non-reproducible due to the lack of publicly available training data. |
| Approach: | They propose a system for long-form song generation with fine-grained style conditioning that includes a licensed synthetic dataset and a song generation model, Muse. |
| Outcome: | The proposed system achieves competitive performance on phoneme error rate, text–music style similarity, and audio aesthetic quality while enabling controllable segment-level generation across different musical structures. |
Copied to clipboard
| Challenge: | Existing studies on multi-modal neural machine translation focus on visual information, but text and image may not match exactly, and visual noise is often ignored. |
| Approach: | They propose a noise-robust multi-modal interactive fusion approach with cross-modal relation-aware mask mechanism for MNMT. |
| Outcome: | The proposed model achieves state-of-the-art scores in all En-De, En-Fr and En-Cs translation tasks. |
Copied to clipboard
| Challenge: | Existing evaluators compress diverse human judgments into a single scalar, leading to brittle alignment and reward hacking. |
| Approach: | They propose a Gaussian-based reinterpretation of reward evaluation as a conditional distribution and a mixture of Gaussians to capture conflicting preference dimensions. |
| Outcome: | The proposed model outperforms scalar baselines in accuracy and generalization. |
Copied to clipboard
| Challenge: | Recent advances in large language models have shown promising ability to perform commonsense reasoning. |
| Approach: | They propose a two-dimensional analysis framework that incorporates token back-tracing and token decoding to uncover how LLMs conduct factual knowledge recall. |
| Outcome: | The proposed framework shows that LLMs lack relevant knowledge but struggle to select the most accurate information based on context during the retrieval and rerank phase. |
Copied to clipboard
| Challenge: | Text-Centric Visual Question Answering (TEC-VQA) is a text-centric visual task understanding tool. |
| Approach: | They introduce a benchmark that features human expert annotations across 9 languages . they prioritize the text in question-answer pairs while disregarding visual text in images . |
| Outcome: | The proposed benchmarks prioritize the text in question-answer pairs while disregarding visual text in images. |
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) show impressive capabilities across visual–language tasks, but their capacity to evaluate artistic expression remains limited. |
| Approach: | They propose an attribute-specific multi-LoRA approach where each attribute corresponds to a distinct evaluation dimension in the scoring rubric. |
| Outcome: | The proposed approach increases correlation from 0.468 to 0.653 on Qwen2.5-VL-7B, with the largest gains on perceptual dimensions and narrowed gaps on higher-order attributes. |
Copied to clipboard
| Challenge: | a tutorial on task instruction is aimed at researchers and practitioners interested in NLP generalization . labeled examples are unlikely to be available in large numbers or do not exist . |
| Approach: | This tutorial will examine the progress of natural language processing (NLP) using labeled examples. authors propose that task instructions act as a novel resource for supervision. |
| Outcome: | This tutorial aims to answer questions about instruction-driven NLP . it focuses on the use of task instructions in a low-shot scenario . |
Copied to clipboard
| Challenge: | Existing frameworks for sequence labeling and classification require massive human effort and labeling data is limited. |
| Approach: | They propose a web-based, Label-Efficient AnnotatioN framework that allows an annotator to provide the needed labels for a task and can capture explanations for each labeling decision. |
| Outcome: | The proposed framework surpasses baseline F1 scores by 5-10 percentage points while using 2X times fewer labeled instances. |
Copied to clipboard
| Challenge: | Existing approaches to reward modeling in reinforcement learning tasks are limited when dealing with ambiguous preferences. |
| Approach: | They propose to use AAM to dynamically calibrate preference margins using the Bradley-Terry model's internal parameter knowledge to improve reward modeling in subjective tasks. |
| Outcome: | The proposed approach improves reward modeling by dynamically calibrating preference margins using the model’s internal parameter knowledge. |
Copied to clipboard
| Challenge: | EmoHarbor is an evaluation framework that rewards generic empathetic responses but fails to assess whether the support is genuinely personalized to users’ unique psychological profiles and contextual needs. |
| Approach: | They propose an automated evaluation framework that adopts a User-as-a-Judge paradigm by simulating the user's inner world. |
| Outcome: | The proposed framework decomposes users' internal processes into three specialized roles and defines 10 evaluation dimensions of personalized support quality. |
Copied to clipboard
| Challenge: | Existing approaches treat instruction-based text editing as a generic text generation problem. Existing methods either over-edit or fail to apply modifications consistently. |
| Approach: | They propose a framework that processes each editing request to best align with it. |
| Outcome: | The proposed framework achieves 9% improvement over the state-of-the-art model. |
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) have impressive multimodal abilities but remain prone to multilingual object hallucination. |
| Approach: | They propose a cross-lingual attention intervention method to mitigate multilingual object hallucination in LVLMs by aligning attention patterns. |
| Outcome: | The proposed method improves 13.56% (up to 30%) on the POPE and 21.75% on the hallucination subsets across languages. |
Copied to clipboard
| Challenge: | Large language models require a balance between efficiency and performance. |
| Approach: | They propose a low-rank compression technique that reduces non-essential parameters by decomposing weight matrices into products of two low-ranked matrici. |
| Outcome: | The proposed method outperforms existing pruning and low-rank compression techniques in maintaining model performance at the same compression ratio. |
Copied to clipboard
| Challenge: | Recent studies show that LLM-based agents exhibit superior moral and emotional language performance compared to humans, raising expectations for their deployment in persuasive tasks. |
| Approach: | They propose a framework for generating persuasive multi-turn dialogues via agent self-play using user agents designed to simulate diverse persona-driven behaviors, a Dialog Agent executing task-oriented persuasion strategies and an Optimization Agent evaluating and refining dialogue outcomes. |
| Outcome: | The proposed framework significantly improved the persuasion capacity of small LLMs, increasing the organic traffic conversion rate by 22.4% (from 1.83% to 2.24%) . |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) show great potential for expressing empathy, but often deliver generic responses that fail to address users’ specific needs. |
| Approach: | They propose a self-evolution framework to help LLMs improve their responses to better align with users’ implicit preferences concerning personality, emotional state, and specific context. |
| Outcome: | The proposed model significantly improves the model's performance in emotional support, reducing unhelpful responses and minimizing discrepancies between user preferences and model outputs. |
Copied to clipboard
| Challenge: | Automated Essay Scoring (AES) systems face three major challenges: reliance on handcrafted features that limit generalizability, difficulty in capturing fine-grained traits like coherence and argumentation, and inability to handle multimodal contexts. |
| Approach: | They propose a multimodal benchmark to evaluate AES capabilities across lexical-, sentence-, and discourse-level traits without manual feature engineering. |
| Outcome: | The proposed system can evaluate AES capabilities across lexical-, sentence-, and discourse-level traits without manual feature engineering. |
Copied to clipboard
| Challenge: | Large Language Models have shown impressive ability to perform in-context learning from only a few examples, but their accuracy varies widely from task to task. |
| Approach: | They propose a method that trains a meta-model using LLM confidence scores as features to perform ICL accuracy estimation. |
| Outcome: | The proposed method improves over baselines across 7 out of 12 settings and achieves the same accuracy as evaluating on 40 sampled examples per task. |
Copied to clipboard
| Challenge: | Large pre-trained models are capable of few-shot in-context learning (ICL) however, concatenated demonstrations are often excessively long and require additional computation. |
| Approach: | They propose to apply fusion-in-decoder (FiD) models to perform few-shot in-context learning (ICL) they propose to use concatenation-based, early-fusion, intermediate- and late-fusion methods to improve efficiency . |
| Outcome: | The proposed methods outperform concatenation-based models on 11 held-out tasks. |
Copied to clipboard
| Challenge: | Existing methods to reduce inference cost by distilling transformer models into lightweight student models are limited for high-volume use cases. |
| Approach: | They propose to distill state-of-the-art transformer models into lightweight student models to reduce computation cost at inference time. |
| Outcome: | The proposed pipeline achieves up to 600x speed-up on GPUs and CPUs on six single-sentence text classification tasks and in domain generalization settings. |
Copied to clipboard
| Challenge: | Recent advances in machine reading comprehension rely heavily on large-scale annotated corpora, which are timeconsuming and costly to collect. |
| Approach: | They propose to use semi-structured explanations to “teach” machines reading comprehension using a small number of semi-structural explanations that explicitly inform machines why answer spans are correct. |
| Outcome: | The proposed method achieves 70.14% F1 score with supervision from 26 explanations on the SQuAD dataset, comparable to plain supervised learning using 1,100 labeled instances yielding a 12x speed up. |
Copied to clipboard
| Challenge: | Existing knowledge extraction tools are not complete due to emerging entities and relations in real-world applications. |
| Approach: | They propose an open-source knowledge extraction toolkit DeepKE that supports low-resource, document-level and multimodal scenarios in the knowledge base population. |
| Outcome: | The proposed toolkit supports low-resource, document-level and multimodal scenarios in the knowledge base population. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated promising potential in providing empathetic support during interactions, but their responses are often verbose or overly formulaic, failing to adequately address the diverse emotional support needs of real-world scenarios. |
| Approach: | They propose a strategy-enhanced role-playing framework that emulates real-world interactions and a dataset that is used to develop an emotional support agent. |
| Outcome: | The proposed framework emulates real-world interactions and promotes a broader range of dialogues and Emotional Support Agent training. |
Copied to clipboard
| Challenge: | Existing approaches to multi-source neural machine translation neglect inconsistencies between sources of information. |
| Approach: | They propose a source invariance network to learn invariant information of parallel sources . they propose to integrate such network with multi-encoder based multi-source NMT methods . |
| Outcome: | The proposed approach achieves clear gains in translation quality and captures implicit invariance between different sources. |
Copied to clipboard
| Challenge: | Existing 3D AIGC methods don’t fully unleash human creativity. |
| Approach: | They propose a framework that generates 3D content from multimodal inputs . they propose 198 multimodal text inputs for 3D generation tasks . |
| Outcome: | The proposed framework generates 3D content from multimodal inputs without human intervention. |
Copied to clipboard
| Challenge: | Pre-trained text-to-text transformers have achieved impressive performance across a range of NLP tasks, such as question answering and commonsense reasoning. |
| Approach: | They propose a framework that improves text-to-text transformer’s generalization ability to unseen tasks by training a hypernetwork to generate task-specific adapters from task descriptions. |
| Outcome: | Experiments on ZEST and a synthetic SQuAD dataset show that Hypter improves upon fine-tuning baselines. |
Copied to clipboard
| Challenge: | Unsupervised Text Style Transfer (UTST) aims to transfer the stylistic properties of a given text without parallel text pairs. |
| Approach: | They propose a SFT-then-PPO paradigm to fine-tune an LLM with parallel data and reward functions for distinguishing stylistic intensity in hierarchical levels. |
| Outcome: | The proposed system can transfer stylistic properties without parallel text pairs even for adjacent levels of intensity. |
Copied to clipboard
| Challenge: | a benchmark is designed to evaluate the capability of Large Multimodal Models (LMMs) in converting complex, structured digital graphics into executable code. |
| Approach: | They propose a benchmark to evaluate the capability of Large Multimodal Models to convert digital graphics into executable code. |
| Outcome: | The proposed benchmark exposes the performance gap among leading LMMs . the benchmark features 1130 meticulously curated samples . |
Copied to clipboard
| Challenge: | Existing studies on DS-based relation extraction (RE) methods focus on handling label noise, but other factors may have been overlooked. |
| Approach: | They propose a method to automatically adjust DS-RE models to a shifted label distribution problem . they find this problem exists in real-world DS datasets and can be overcome . |
| Outcome: | The proposed method achieves consistent performance gains on DS-trained models with an up to 23% relative F1 improvement, which verifies their assumptions. |
Copied to clipboard
| Challenge: | We study whether and how cross-task generalization ability can be acquired . we use CrossFit to standardize seen/unseen task partitions and evaluation protocols . |
| Approach: | They propose a problem setup for studying cross-task generalization ability which standardizes seen/unseen task partitions and data access during different learning stages. |
| Outcome: | The proposed model can be used to build few-shot learners across diverse tasks. |
Copied to clipboard
| Challenge: | Current methods of creating accessible movies rely on manual work, resulting in high costs and limited scalability. |
| Approach: | They propose a multi-modal movie audio description pipeline that generates narrations of information that is not accessible through unimodal hearing in movies. |
| Outcome: | The proposed pipeline surpasses existing baselines in performance on widely used datasets. |
Copied to clipboard
| Challenge: | Current methods struggle to distinguish targets in low Signal-to-Noise Ratio environments and lack sufficient pre-execution verification to prevent error accumulation. |
| Approach: | They propose a Memory-augmented Debate System to ensure precise grounding across diverse interfaces and handle irreversible errors in extended workflows. |
| Outcome: | The proposed system achieves a 90.23% task success rate on MaDS-Benchmark and strong performance on public benchmarks including AITW, AITZ, CAGUI, and GUIOdyssey. |
Copied to clipboard
| Challenge: | Existing studies show that inserting an intermediate pre-training stage improves performance of masked language models. |
| Approach: | They propose methods to automate the discovery of optimal masking policies via direct supervision or meta-learning. |
| Outcome: | The proposed method outperforms the heuristic of masking named entities on TriviaQA and can be generalizable beyond that task. |
Copied to clipboard
| Challenge: | a recent study shows that large language models can be used to predict performance on new configurations. |
| Approach: | They investigate the predictability of large language model capabilities by using BIG-bench . they find a subset of BIG-Bench tasks as informative as BIG-bnch Hard . |
| Outcome: | The proposed model achieves an R2 score greater than 95% on BIG-bench . the model is 3 smaller than BIG-Bench Hard, and the model performs better on the full set. |
Copied to clipboard
| Challenge: | Existing methods for expressive text-to-speech only implicitly learn prosody with masked token reconstruction tasks. |
| Approach: | They propose a cross-modal contrastive pre-training framework that learns from prosody variance of the same text token under different contexts. |
| Outcome: | The proposed framework can learn from prosody variance of a text token under different contexts. |
Copied to clipboard
| Challenge: | Expressive zero-shot voice conversion (VC) aims to modify source timbre to match unseen speaker . existing zero- shot VC systems struggle to reproduce paralinguistic information in highly expressive speech . |
| Approach: | They propose a framework for expressive zero-shot voice conversion that uses hybrid content encoding and memory-augmented context-aware timbre modeling. |
| Outcome: | The proposed framework surpasses state-of-the-art VC systems in speech naturalness, speaker similarity, and speaker similarness. |
Copied to clipboard
| Challenge: | Existing models for speech-to-speech translation suffer from distinct degradation in noisy environments and fail to translate visual speech. |
| Approach: | They propose a text-based audio-visual speech-to-speech translation model that integrates visual information with audio-only data to improve system robustness. |
| Outcome: | The proposed model outperforms models trained on audio-only corpus in two languages . it also improves with low-resource audio-visual data, compared with baselines . |
Copied to clipboard
| Challenge: | Existing methods for relation detection only detect one path to obtain the answer without considering other correct paths. |
| Approach: | They propose a divide-and-conquer approach for multi-label multi-hop relation detection . they propose 'path sampling mechanism' to generate diverse relation paths . |
| Outcome: | The proposed approach outperforms other competitive approaches on the FreebaseQA benchmark dataset. |
Copied to clipboard
| Challenge: | Existing studies on biases within specific domains, such as finance, remain limited. |
| Approach: | They propose a framework to detect, detect, analyze and mitigate financial biases in large language models. |
| Outcome: | The proposed framework reduces bias by 68% for the most biased model, according to key metrics. |
Copied to clipboard
| Challenge: | Adapting existing approaches for converting natural language to SQL encounters hurdles due to distinct nature of GQL compared to SQL. |
| Approach: | They propose a method that integrates both small and large Foundation Models for ranking, rewriting, and refining tasks. |
| Outcome: | The proposed approach integrates both small and large Foundation Models for ranking, rewriting, and refining tasks while capitalizing on the superior generalization and query generation prowess of larger models for the final transformation of natural language queries into GQL formats. |
Copied to clipboard
| Challenge: | Pre-trained transformer models are capable of multitasking on diverse NLP tasks, but little is known about how multitaskability and cross-task generalization is achieved. |
| Approach: | They propose to use a transformer-based mixture-of-expert model with a router component to choose among experts dynamically and flexibly. |
| Outcome: | The proposed models improve the average performance gain (ARG) metric by 2.6% when adapting to unseen tasks, and by 5.6% in zero-shot generalization settings. |