Papers by Xiang Ye

40 papers
Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Control (2026.findings-acl)

Copied to clipboard

Challenge: Recent commercial systems such as Suno demonstrate strong capabilities in long-form song generation, but academic research remains non-reproducible due to the lack of publicly available training data.
Approach: They propose a system for long-form song generation with fine-grained style conditioning that includes a licensed synthetic dataset and a song generation model, Muse.
Outcome: The proposed system achieves competitive performance on phoneme error rate, text–music style similarity, and audio aesthetic quality while enabling controllable segment-level generation across different musical structures.
Noise-robust Cross-modal Interactive Learning with Text2Image Mask for Multi-modal Neural Machine Translation (2022.coling-1)

Copied to clipboard

Challenge: Existing studies on multi-modal neural machine translation focus on visual information, but text and image may not match exactly, and visual noise is often ignored.
Approach: They propose a noise-robust multi-modal interactive fusion approach with cross-modal relation-aware mask mechanism for MNMT.
Outcome: The proposed model achieves state-of-the-art scores in all En-De, En-Fr and En-Cs translation tasks.
PRISM: Probabilistic Reward Model with Inherent Structural Modeling (2026.acl-long)

Copied to clipboard

Challenge: Existing evaluators compress diverse human judgments into a single scalar, leading to brittle alignment and reward hacking.
Approach: They propose a Gaussian-based reinterpretation of reward evaluation as a conditional distribution and a mixture of Gaussians to capture conflicting preference dimensions.
Outcome: The proposed model outperforms scalar baselines in accuracy and generalization.
Tracing and Dissecting How LLMs Recall Factual Knowledge for Real World Questions (2025.acl-long)

Copied to clipboard

Challenge: Recent advances in large language models have shown promising ability to perform commonsense reasoning.
Approach: They propose a two-dimensional analysis framework that incorporates token back-tracing and token decoding to uncover how LLMs conduct factual knowledge recall.
Outcome: The proposed framework shows that LLMs lack relevant knowledge but struggle to select the most accurate information based on context during the retrieval and rerank phase.
MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering (2025.findings-acl)

Copied to clipboard

Challenge: Text-Centric Visual Question Answering (TEC-VQA) is a text-centric visual task understanding tool.
Approach: They introduce a benchmark that features human expert annotations across 9 languages . they prioritize the text in question-answer pairs while disregarding visual text in images .
Outcome: The proposed benchmarks prioritize the text in question-answer pairs while disregarding visual text in images.
KidsArtBench: Multi-Dimensional Children’s Art Evaluation with Attribute-Aware MLLMs (2026.eacl-long)

Copied to clipboard

Challenge: Multimodal Large Language Models (MLLMs) show impressive capabilities across visual–language tasks, but their capacity to evaluate artistic expression remains limited.
Approach: They propose an attribute-specific multi-LoRA approach where each attribute corresponds to a distinct evaluation dimension in the scoring rubric.
Outcome: The proposed approach increases correlation from 0.468 to 0.653 on Qwen2.5-VL-7B, with the largest gains on perceptual dimensions and narrowed gaps on higher-order attributes.
LLM-driven Instruction Following: Progresses and Concerns (2023.emnlp-tutorial)

Copied to clipboard

Challenge: a tutorial on task instruction is aimed at researchers and practitioners interested in NLP generalization . labeled examples are unlikely to be available in large numbers or do not exist .
Approach: This tutorial will examine the progress of natural language processing (NLP) using labeled examples. authors propose that task instructions act as a novel resource for supervision.
Outcome: This tutorial aims to answer questions about instruction-driven NLP . it focuses on the use of task instructions in a low-shot scenario .
LEAN-LIFE: A Label-Efficient Annotation Framework Towards Learning from Explanation (2020.acl-demos)

Copied to clipboard

Challenge: Existing frameworks for sequence labeling and classification require massive human effort and labeling data is limited.
Approach: They propose a web-based, Label-Efficient AnnotatioN framework that allows an annotator to provide the needed labels for a task and can capture explanations for each labeling decision.
Outcome: The proposed framework surpasses baseline F1 scores by 5-10 percentage points while using 2X times fewer labeled instances.
Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity (2026.acl-long)

Copied to clipboard

Challenge: Existing approaches to reward modeling in reinforcement learning tasks are limited when dealing with ambiguous preferences.
Approach: They propose to use AAM to dynamically calibrate preference margins using the Bradley-Terry model's internal parameter knowledge to improve reward modeling in subjective tasks.
Outcome: The proposed approach improves reward modeling by dynamically calibrating preference margins using the model’s internal parameter knowledge.
EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User’s Internal World (2026.acl-long)

Copied to clipboard

Challenge: EmoHarbor is an evaluation framework that rewards generic empathetic responses but fails to assess whether the support is genuinely personalized to users’ unique psychological profiles and contextual needs.
Approach: They propose an automated evaluation framework that adopts a User-as-a-Judge paradigm by simulating the user's inner world.
Outcome: The proposed framework decomposes users' internal processes into three specialized roles and defines 10 evaluation dimensions of personalized support quality.
HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworks (2026.findings-acl)

Copied to clipboard

Challenge: Existing approaches treat instruction-based text editing as a generic text generation problem. Existing methods either over-edit or fail to apply modifications consistently.
Approach: They propose a framework that processes each editing request to best align with it.
Outcome: The proposed framework achieves 9% improvement over the state-of-the-art model.
CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention (2025.acl-long)

Copied to clipboard

Challenge: Large Vision-Language Models (LVLMs) have impressive multimodal abilities but remain prone to multilingual object hallucination.
Approach: They propose a cross-lingual attention intervention method to mitigate multilingual object hallucination in LVLMs by aligning attention patterns.
Outcome: The proposed method improves 13.56% (up to 30%) on the POPE and 21.75% on the hallucination subsets across languages.
Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization (2024.findings-emnlp)

Copied to clipboard

Challenge: Large language models require a balance between efficiency and performance.
Approach: They propose a low-rank compression technique that reduces non-essential parameters by decomposing weight matrices into products of two low-ranked matrici.
Outcome: The proposed method outperforms existing pruning and low-rank compression techniques in maintaining model performance at the same compression ratio.
MADS: Multi-Agent Dialogue Simulation for Diverse Persuasion Data Generation (2025.emnlp-industry)

Copied to clipboard

Challenge: Recent studies show that LLM-based agents exhibit superior moral and emotional language performance compared to humans, raising expectations for their deployment in persuasive tasks.
Approach: They propose a framework for generating persuasive multi-turn dialogues via agent self-play using user agents designed to simulate diverse persona-driven behaviors, a Dialog Agent executing task-oriented persuasion strategies and an Optimization Agent evaluating and refining dialogue outcomes.
Outcome: The proposed framework significantly improved the persuasion capacity of small LLMs, increasing the organic traffic conversion rate by 22.4% (from 1.83% to 2.24%) .
From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignment (2025.findings-emnlp)

Copied to clipboard

Challenge: Large Language Models (LLMs) show great potential for expressing empathy, but often deliver generic responses that fail to address users’ specific needs.
Approach: They propose a self-evolution framework to help LLMs improve their responses to better align with users’ implicit preferences concerning personality, emotional state, and specific context.
Outcome: The proposed model significantly improves the model's performance in emotional support, reducing unhelpful responses and minimizing discrepancies between user preferences and model outputs.
EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models (2025.findings-acl)

Copied to clipboard

Challenge: Automated Essay Scoring (AES) systems face three major challenges: reliance on handcrafted features that limit generalizability, difficulty in capturing fine-grained traits like coherence and argumentation, and inability to handle multimodal contexts.
Approach: They propose a multimodal benchmark to evaluate AES capabilities across lexical-, sentence-, and discourse-level traits without manual feature engineering.
Outcome: The proposed system can evaluate AES capabilities across lexical-, sentence-, and discourse-level traits without manual feature engineering.
Estimating Large Language Model Capabilities without Labeled Test Data (2023.findings-emnlp)

Copied to clipboard

Challenge: Large Language Models have shown impressive ability to perform in-context learning from only a few examples, but their accuracy varies widely from task to task.
Approach: They propose a method that trains a meta-model using LLM confidence scores as features to perform ICL accuracy estimation.
Outcome: The proposed method improves over baselines across 7 out of 12 settings and achieves the same accuracy as evaluating on 40 sampled examples per task.
FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning (2023.acl-long)

Copied to clipboard

Challenge: Large pre-trained models are capable of few-shot in-context learning (ICL) however, concatenated demonstrations are often excessively long and require additional computation.
Approach: They propose to apply fusion-in-decoder (FiD) models to perform few-shot in-context learning (ICL) they propose to use concatenation-based, early-fusion, intermediate- and late-fusion methods to improve efficiency .
Outcome: The proposed methods outperform concatenation-based models on 11 held-out tasks.
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models (2022.naacl-main)

Copied to clipboard

Challenge: Existing methods to reduce inference cost by distilling transformer models into lightweight student models are limited for high-volume use cases.
Approach: They propose to distill state-of-the-art transformer models into lightweight student models to reduce computation cost at inference time.
Outcome: The proposed pipeline achieves up to 600x speed-up on GPUs and CPUs on six single-sentence text classification tasks and in domain generalization settings.
Teaching Machine Comprehension with Compositional Explanations (2020.findings-emnlp)

Copied to clipboard

Challenge: Recent advances in machine reading comprehension rely heavily on large-scale annotated corpora, which are timeconsuming and costly to collect.
Approach: They propose to use semi-structured explanations to “teach” machines reading comprehension using a small number of semi-structural explanations that explicitly inform machines why answer spans are correct.
Outcome: The proposed method achieves 70.14% F1 score with supervision from 26 explanations on the SQuAD dataset, comparable to plain supervised learning using 1,100 labeled instances yielding a 12x speed up.
DeepKE: A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base Population (2022.emnlp-demos)

Copied to clipboard

Challenge: Existing knowledge extraction tools are not complete due to emerging entities and relations in real-world applications.
Approach: They propose an open-source knowledge extraction toolkit DeepKE that supports low-resource, document-level and multimodal scenarios in the knowledge base population.
Outcome: The proposed toolkit supports low-resource, document-level and multimodal scenarios in the knowledge base population.
SweetieChat: A Strategy-Enhanced Role-playing Framework for Diverse Scenarios Handling Emotional Support Agent (2025.coling-main)

Copied to clipboard

Challenge: Large Language Models (LLMs) have demonstrated promising potential in providing empathetic support during interactions, but their responses are often verbose or overly formulaic, failing to adequately address the diverse emotional support needs of real-world scenarios.
Approach: They propose a strategy-enhanced role-playing framework that emulates real-world interactions and a dataset that is used to develop an emotional support agent.
Outcome: The proposed framework emulates real-world interactions and promotes a broader range of dialogues and Emotional Support Agent training.
Meet Changes with Constancy: Learning Invariance in Multi-Source Translation (2020.coling-main)

Copied to clipboard

Challenge: Existing approaches to multi-source neural machine translation neglect inconsistencies between sources of information.
Approach: They propose a source invariance network to learn invariant information of parallel sources . they propose to integrate such network with multi-encoder based multi-source NMT methods .
Outcome: The proposed approach achieves clear gains in translation quality and captures implicit invariance between different sources.
Idea23D: Collaborative LMM Agents Enable 3D Model Generation from Interleaved Multimodal Inputs (2025.coling-main)

Copied to clipboard

Challenge: Existing 3D AIGC methods don’t fully unleash human creativity.
Approach: They propose a framework that generates 3D content from multimodal inputs . they propose 198 multimodal text inputs for 3D generation tasks .
Outcome: The proposed framework generates 3D content from multimodal inputs without human intervention.
Learning to Generate Task-Specific Adapters from Task Description (2021.acl-short)

Copied to clipboard

Challenge: Pre-trained text-to-text transformers have achieved impressive performance across a range of NLP tasks, such as question answering and commonsense reasoning.
Approach: They propose a framework that improves text-to-text transformer’s generalization ability to unseen tasks by training a hypernetwork to generate task-specific adapters from task descriptions.
Outcome: Experiments on ZEST and a synthetic SQuAD dataset show that Hypter improves upon fine-tuning baselines.
Unsupervised Text Style Transfer for Controllable Intensity (2026.findings-eacl)

Copied to clipboard

Challenge: Unsupervised Text Style Transfer (UTST) aims to transfer the stylistic properties of a given text without parallel text pairs.
Approach: They propose a SFT-then-PPO paradigm to fine-tune an LLM with parallel data and reward functions for distinguishing stylistic intensity in hierarchical levels.
Outcome: The proposed system can transfer stylistic properties without parallel text pairs even for adjacent levels of intensity.
Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation (2026.acl-long)

Copied to clipboard

Challenge: a benchmark is designed to evaluate the capability of Large Multimodal Models (LMMs) in converting complex, structured digital graphics into executable code.
Approach: They propose a benchmark to evaluate the capability of Large Multimodal Models to convert digital graphics into executable code.
Outcome: The proposed benchmark exposes the performance gap among leading LMMs . the benchmark features 1130 meticulously curated samples .
Looking Beyond Label Noise: Shifted Label Distribution Matters in Distantly Supervised Relation Extraction (D19-1)

Copied to clipboard

Challenge: Existing studies on DS-based relation extraction (RE) methods focus on handling label noise, but other factors may have been overlooked.
Approach: They propose a method to automatically adjust DS-RE models to a shifted label distribution problem . they find this problem exists in real-world DS datasets and can be overcome .
Outcome: The proposed method achieves consistent performance gains on DS-trained models with an up to 23% relative F1 improvement, which verifies their assumptions.
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021.emnlp-main)

Copied to clipboard

Challenge: We study whether and how cross-task generalization ability can be acquired . we use CrossFit to standardize seen/unseen task partitions and evaluation protocols .
Approach: They propose a problem setup for studying cross-task generalization ability which standardizes seen/unseen task partitions and data access during different learning stages.
Outcome: The proposed model can be used to build few-shot learners across diverse tasks.
MMAD:Multi-modal Movie Audio Description (2024.lrec-main)

Copied to clipboard

Challenge: Current methods of creating accessible movies rely on manual work, resulting in high costs and limited scalability.
Approach: They propose a multi-modal movie audio description pipeline that generates narrations of information that is not accessible through unimodal hearing in movies.
Outcome: The proposed pipeline surpasses existing baselines in performance on widely used datasets.
MaDS: Long-Horizon GUI Automation via Synergizing Dual-Layer Memory and Multi-Round Debate (2026.acl-long)

Copied to clipboard

Challenge: Current methods struggle to distinguish targets in low Signal-to-Noise Ratio environments and lack sufficient pre-execution verification to prevent error accumulation.
Approach: They propose a Memory-augmented Debate System to ensure precise grounding across diverse interfaces and handle irreversible errors in extended workflows.
Outcome: The proposed system achieves a 90.23% task success rate on MaDS-Benchmark and strong performance on public benchmarks including AITW, AITZ, CAGUI, and GUIOdyssey.
On the Influence of Masking Policies in Intermediate Pre-training (2021.emnlp-main)

Copied to clipboard

Challenge: Existing studies show that inserting an intermediate pre-training stage improves performance of masked language models.
Approach: They propose methods to automate the discovery of optimal masking policies via direct supervision or meta-learning.
Outcome: The proposed method outperforms the heuristic of masking named entities on TriviaQA and can be generalizable beyond that task.
How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench (2023.findings-emnlp)

Copied to clipboard

Challenge: a recent study shows that large language models can be used to predict performance on new configurations.
Approach: They investigate the predictability of large language model capabilities by using BIG-bench . they find a subset of BIG-Bench tasks as informative as BIG-bnch Hard .
Outcome: The proposed model achieves an R2 score greater than 95% on BIG-bench . the model is 3 smaller than BIG-Bench Hard, and the model performs better on the full set.
CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-Training (2023.acl-long)

Copied to clipboard

Challenge: Existing methods for expressive text-to-speech only implicitly learn prosody with masked token reconstruction tasks.
Approach: They propose a cross-modal contrastive pre-training framework that learns from prosody variance of the same text token under different contexts.
Outcome: The proposed framework can learn from prosody variance of a text token under different contexts.
Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modeling (2025.acl-long)

Copied to clipboard

Challenge: Expressive zero-shot voice conversion (VC) aims to modify source timbre to match unseen speaker . existing zero- shot VC systems struggle to reproduce paralinguistic information in highly expressive speech .
Approach: They propose a framework for expressive zero-shot voice conversion that uses hybrid content encoding and memory-augmented context-aware timbre modeling.
Outcome: The proposed framework surpasses state-of-the-art VC systems in speech naturalness, speaker similarity, and speaker similarness.
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation (2023.acl-long)

Copied to clipboard

Challenge: Existing models for speech-to-speech translation suffer from distinct degradation in noisy environments and fail to translate visual speech.
Approach: They propose a text-based audio-visual speech-to-speech translation model that integrates visual information with audio-only data to improve system robustness.
Outcome: The proposed model outperforms models trained on audio-only corpus in two languages . it also improves with low-resource audio-visual data, compared with baselines .
A Divide-And-Conquer Approach for Multi-label Multi-hop Relation Detection in Knowledge Base Question Answering (2021.findings-emnlp)

Copied to clipboard

Challenge: Existing methods for relation detection only detect one path to obtain the answer without considering other correct paths.
Approach: They propose a divide-and-conquer approach for multi-label multi-hop relation detection . they propose 'path sampling mechanism' to generate diverse relation paths .
Outcome: The proposed approach outperforms other competitive approaches on the FreebaseQA benchmark dataset.
Are LLMs Rational Investors? A Study on the Financial Bias in LLMs (2025.findings-acl)

Copied to clipboard

Challenge: Existing studies on biases within specific domains, such as finance, remain limited.
Approach: They propose a framework to detect, detect, analyze and mitigate financial biases in large language models.
Outcome: The proposed framework reduces bias by 68% for the most biased model, according to key metrics.
R3-NL2GQL: A Model Coordination and Knowledge Graph Alignment Approach for NL2GQL (2024.findings-emnlp)

Copied to clipboard

Challenge: Adapting existing approaches for converting natural language to SQL encounters hurdles due to distinct nature of GQL compared to SQL.
Approach: They propose a method that integrates both small and large Foundation Models for ranking, rewriting, and refining tasks.
Outcome: The proposed approach integrates both small and large Foundation Models for ranking, rewriting, and refining tasks while capitalizing on the superior generalization and query generation prowess of larger models for the final transformation of natural language queries into GQL formats.
Eliciting and Understanding Cross-task Skills with Task-level Mixture-of-Experts (2022.findings-emnlp)

Copied to clipboard

Challenge: Pre-trained transformer models are capable of multitasking on diverse NLP tasks, but little is known about how multitaskability and cross-task generalization is achieved.
Approach: They propose to use a transformer-based mixture-of-expert model with a router component to choose among experts dynamically and flexibly.
Outcome: The proposed models improve the average performance gain (ARG) metric by 2.6% when adapting to unseen tasks, and by 5.6% in zero-shot generalization settings.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations