Papers by Ji Qi
Copied to clipboard
| Challenge: | Recent commercial systems such as Suno demonstrate strong capabilities in long-form song generation, but academic research remains non-reproducible due to the lack of publicly available training data. |
| Approach: | They propose a system for long-form song generation with fine-grained style conditioning that includes a licensed synthetic dataset and a song generation model, Muse. |
| Outcome: | The proposed system achieves competitive performance on phoneme error rate, text–music style similarity, and audio aesthetic quality while enabling controllable segment-level generation across different musical structures. |
Copied to clipboard
| Challenge: | Existing unsupervised methods for paraphrase generation are weak in semantic equivalence or expression diversity. |
| Approach: | They propose a framework for unsupervised paraphrase generation that employs multi-aspect equivalence constraints and multi-granularity diversifying mechanisms to achieve good semantic equvalence and expressive diversity. |
| Outcome: | The proposed framework achieves 9.1% and 3.3% absolute gains over previous SOTA on Quora and MSCOCO and can improve to 18.0% and 4.6% on GLUE. |
Copied to clipboard
| Challenge: | We propose a new task to extract events and their arguments from multimedia documents . traditional methods target text, images or videos, but multimedia content is distributed via multimedia . |
| Approach: | They propose a method that encodes structured representations of semantic information from textual and visual data into a common embedding space. |
| Outcome: | The proposed method achieves 4.0% and 9.8% absolute gains on text event argument role labeling and visual event extraction. |
Copied to clipboard
| Challenge: | Several studies rely on additional models to optimize mixtures. |
| Approach: | They propose a method that dynamically optimizes instruction-tuning dataset mixtures by prior-scaled Boltzmann Exploration and a multi-armed bandit setup. |
| Outcome: | The proposed method improves the TÜLU-2-mixture and TÜLO-3-mixtures across 10 benchmarks while introducing minimal computational overhead over naive sampling. |
Copied to clipboard
| Challenge: | Existing benchmarks evaluate agents in simplified, idealized settings, relying on pre-packaged tool interfaces, overlooking critical steps, and assume inputs are clean and fully specified. |
| Approach: | They propose a framework that evaluates language agents in simplified, idealized settings . they show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
| Outcome: | Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
Copied to clipboard
| Challenge: | Existing approaches to online claim verification rely on prompt engineering or pre-designed reasoning workflows. |
| Approach: | They propose an online reinforcement learning framework that enables an LLM to interact with a search engine and receive reward signals that explicitly shape its planning, retrieval, and reasoning behaviors. |
| Outcome: | Empirical results show that Veri-R1 improves joint accuracy by 30% and doubles evidence score, often surpassing larger-scale model counterparts. |
Copied to clipboard
| Challenge: | Besides Transformers without position encodings, the success of NoPE provides a new way to overcome the challenge of generalizing to longer sentences. |
| Approach: | They propose a parameter-efficient tuning for searching attention heads’ best temperature hyper-parameters, which substantially expands NoPE’s context size. |
| Outcome: | The proposed tuning significantly expands NoPE's context size, allowing it to generalize to longer sentences with state-of-the-art generalization algorithms. |
Copied to clipboard
| Challenge: | First-order logic (FOL) is often used to represent logical entailment, but determining natural language (NL) enanglement using FOL remains a challenge. |
| Approach: | They propose an Entailment-Preserving FOL representations task and a method which trains an NL-to-FOL translator by using the natural language entailment labels as verifiable rewards. |
| Outcome: | The proposed method achieves 1.8–2.7% improvement in EPR and 17.4–20.6% increase in E PR@16 compared to baselines in three datasets. |
Copied to clipboard
| Challenge: | Existing frameworks for detecting fake news videos are limited . a new approach is proposed to integrate neighborhood information of new videos . |
| Approach: | They propose a framework for automatically detecting fake news videos . it integrates neighborhood relationship of new videos belonging to same event . |
| Outcome: | The proposed framework improves performance of existing detectors and graph aggregation and debunking rectification modules. |
Copied to clipboard
| Challenge: | Existing evaluation benchmarks focus on pairwise matching, ignoring robustness . current models exhibit frustrating degradation, with a maximum drop of 23.43 F1 score . |
| Approach: | They propose a benchmark that simulates the evaluation of open information extraction models in the real world . they perform experiments on typical models published in the last decade and a representative large language model . |
| Outcome: | The proposed model is rated robust on a knowledge-invariant clique with different syntactic and expressive forms. |
Copied to clipboard
| Challenge: | Spoken language understanding (SLU) involves intent determination and slot filling . existing joint learning methods only consider joint learning by sharing parameters on surface level rather than semantic level. |
| Approach: | They propose a self-attentive model to fully utilize the semantic correlation between slot and intent. |
| Outcome: | The proposed model outperforms existing methods in both intent detection and slot filling tasks on ATIS benchmarks. |
Copied to clipboard
| Challenge: | State-of-the-art adversarial examples lack a common definition of what constitutes success . human surveys show that to preserve semantics, we need to increase the minimum cosine similarities between the embeddings of swapped words and between the sentence encodings of original and perturbed sentences. |
| Approach: | They propose a unified definition of what constitutes a successful adversarial example . they propose four categories of constraints that are used to define adversarials . |
| Outcome: | The proposed framework is based on the outputs of two state-of-the-art synonym substitution attacks. |
Copied to clipboard
| Challenge: | Text-to-Video (T2V) generation is a challenge under complex scenarios. |
| Approach: | They propose a scenario-aware and self-correcting multi-agent prompt refinement framework for T2V prompting. |
| Outcome: | The proposed framework improves text-to-video alignment and overall generation quality under complex scenarios. |
Copied to clipboard
| Challenge: | Recent research has shown that black-box testing is not applicable to NLP models. |
| Approach: | They propose a set of white-box testing methods that are customized for transformer-based NLP models and adapt them to a black-box test suite. |
| Outcome: | The proposed methods can reduce testing suites by 60% while retaining failing tests, thereby concentrating faultdetection power of the test suite. |
Copied to clipboard
| Challenge: | Large language models struggle to process lengthy inputs due to limited length generalization and attention’s quadratic computational demands. |
| Approach: | They propose a training-free framework that allows each head to attend to important context chunks instead of allowing each head a full sentence . |
| Outcome: | The proposed framework unlocks multi-head attention's untapped potential by allowing each head to attend to important context chunks instead of the full sentence. |
Copied to clipboard
| Challenge: | Existing methods for event prediction are incomplete and noisy. |
| Approach: | They propose to use news-related event schemas to extract newsworthy events . they build a demo website and include a video demonstrating the framework . |
| Outcome: | The proposed framework can be applied to a wide variety of newsworthy scenarios. |
Copied to clipboard
| Challenge: | Current abstractive summarization models generate inconsistent content due to the inherently noisy dataset and the discrepancy between maximum likelihood estimation based training objectives and consistency measurements. |
| Approach: | They propose a new consistency taxonomy that categorizes inconsistent content into faithfulness, factuality, and self-supportiveness. |
| Outcome: | Experiments on XSUM and CNN/DM datasets show that EnergySum mitigates the trade-off between accuracy and consistency. |
Copied to clipboard
| Challenge: | Existing studies for visually-situated language understanding have shown shallow zero-shot visual text recognition ability when fed a low-resolution image with salient text information. |
| Approach: | They propose a model for universal OCR-free visually-situated language understanding based on the Multimodal Large Language Model (MLLM) their model is jointly finetuned on a wide range of visually situated language understanding tasks via a unified instruction format. |
| Outcome: | The proposed model achieves state-of-the-art ocr-free performance in 8 out of 10 visually-situated language understanding tasks across 5 domains: documents, tables, charts, natural images, and webpage screenshots. |
Copied to clipboard
| Challenge: | Existing methods for relation classification have been used in natural language processing. |
| Approach: | They propose a relation classification task for Chinese literature text using a new dataset. |
| Outcome: | The proposed model outperforms the state-of-the-art methods on Chinese literature text. |
Copied to clipboard
| Challenge: | Existing methods to automate event extraction focus on uncertainty, re-occurring events and multiple hypotheses. |
| Approach: | They propose a new Event Graph Schema where two event types are connected through multiple paths involving entities that fill important roles in a coherent story. |
| Outcome: | The proposed model is highly effective at inducing salient and coherent schemas. |
Copied to clipboard
| Challenge: | Existing research on monological argumentation covers claims generation, argument structure prediction, and essay scoring. |
| Approach: | They propose to identify argument pairs from two posts with opposite stances to a certain topic. |
| Outcome: | The proposed framework outperforms competing models on a large-scale dataset . it also proves that it is useful for analyzing argument pairs from two posts . |
Copied to clipboard
| Challenge: | Existing methods for concept-level grounding and instruction-level reasoning use coarse representations and iterative mask filtering. |
| Approach: | They propose an instruction-following extension of the Segment Anything Model 3 family that unifies concept-level grounding and instruction-level reasoning within a single segmentation framework. |
| Outcome: | Experiments show that SAM3-I achieves appealing performance across referring and reasoning-based segmentation while maintaining its strong concept recall ability. |
Copied to clipboard
| Challenge: | Existing methods to train LLMs suffer from overthinking, leading to lengthy reasoning traces . Existing approaches to train large language models suffer from this problem . |
| Approach: | They propose a method to combine multiple reasoning chains for training LLMs . they use stepwise exploration and long-short switched sampling to evaluate reasoning paths . |
| Outcome: | The proposed method reduces reasoning lengths by approximately 30-50% . it also maintains or improves reasoning accuracy compared to baselines . |
Copied to clipboard
| Challenge: | Existing factual consistency evaluation approaches for text summarization provide binary predictions and limited insights into the weakness of summarizing systems. |
| Approach: | They propose a task of fine-grained inconsistency detection to detect inconsistent summaries . they represent facts with semantic frames extracted by semantic role labeling and highlight related semantic frames to predict inconsistance . |
| Outcome: | The proposed model outperforms baseline models and provides evidence to support or refute the summary. |
Copied to clipboard
| Challenge: | despite their extensive context window, long-context language models fail in some basic cases . a recent study shows that long-cot methods are not necessary for long-constituency tasks . |
| Approach: | a new study evaluates long-context language models with a large context window . the authors propose a method that can be well addressed with arbitrary reasoning steps . |
| Outcome: | The proposed methods are well addressed with a sufficient number of reasoning steps, guided by specific CoT prompts. |
Copied to clipboard
| Challenge: | Existing evaluations of tool learning focus on validation of tools for large language models with expected outcomes, but this focus ignores the complex capabilities required for LLMs to effectively use tools. |
| Approach: | They propose a fine-grained system for evaluation of large language models’ tool learning capabilities in authentic scenarios. |
| Outcome: | The proposed system examines seven real-world scenarios, analyzing five dimensions crucial to LLMs in tool learning: format alignment, intent comprehension, behavior planning, tool selection, and answer organization. |
Copied to clipboard
| Challenge: | Goal-oriented script planning is used by humans to plan for typical activities . however, this capability remains underexplored due to several challenges . |
| Approach: | They propose a framework that enables product-enriched scripts by associating products with each step based on the semantic similarity between the actions and their purchase intentions. |
| Outcome: | The proposed framework can generate product-enriched scripts from 2.4 million scripts . human annotations are conducted to provide gold labels for a sampled subset . |
Copied to clipboard
| Challenge: | Open Information Extraction models have shown promising results with sufficient supervision, but the syntactic distribution of training data is partially observable in comparison to the real world. |
| Approach: | They propose a syntactically robust training framework that enables models to be trained on a multi-paraphrase distribution based on diverse paraphrase generation. |
| Outcome: | The proposed framework can be applied to other syntactic partial observable domains. |
Copied to clipboard
| Challenge: | Existing models for long sequences are not efficient due to the quadratic space and time complexity of the self-attention modules. |
| Approach: | They propose to reduce the quadratic complexity to linear (modulo logarithmic factors) by low-dimensional projection and row selection. |
| Outcome: | The proposed methods outperform transformer-based models with smaller time/space footprint on the Long Range Arena benchmark. |
Copied to clipboard
| Challenge: | Existing methods for visual storytelling construct text description independently for each image and roughly concatenate them as a story, which leads to the problem of generating semantically incoherent content. |
| Approach: | They propose a topic description task to detect the global semantic context of an image stream and a story is then constructed with the guidance of the topic description. |
| Outcome: | The proposed framework can generate stories with higher quality compared to state-of-the-art methods on a VIST dataset. |
Copied to clipboard
| Challenge: | Existing benchmarks for multimodal reasoning in large multimodal models are underperforming on multimodal tasks. |
| Approach: | They propose a benchmark for multimodal reasoning in large multimodal models, MM-MATH . MM's process evaluation employs LMM-as-a-judge to automatically analyze solution steps . diagram misinterpretation is the most common error, they find . |
| Outcome: | The proposed model achieves only 31% accuracy, compared to 82% for humans. |
Copied to clipboard
| Challenge: | Existing research explores different text features of reply comments on word level and ignores interactions between participants. |
| Approach: | They propose a co-attention mechanism based neural network to capture interactions between participants on argument level to better model dialogical argumentation. |
| Outcome: | The proposed model outperforms state-of-the-art methods on a publicly available dataset showing that it extracts interactive argument pairs from the original post and the reply. |
Copied to clipboard
| Challenge: | Existing studies to build long context language models focus on context extension and continual training on long text. |
| Approach: | They propose a recipe for instruction fine-tuning on input sequences of similar length . they adopt packing and sorted batching strategies to speed up supervised fine-uning . |
| Outcome: | The proposed model outperforms existing recipes for LLMs in long context tasks by 30% while maintaining proficiency in handling short, generic tasks. |
Copied to clipboard
| Challenge: | Existing work integrates reinforcement learning with compiler feedback to enhance code generation quality but the long code generated by LLMs makes RL exploration ineffective. |
| Approach: | They propose a framework that integrates reinforcement learning and compiler feedback to enhance code generation quality. |
| Outcome: | The proposed framework outperforms state-of-the-art approaches in corresponding benchmarks and integrates reinforcement learning with compiler feedback to improve code generation quality. |
Copied to clipboard
| Challenge: | Multi-head Latent Attention (MLA) is an innovative architecture designed to ensure efficient and economical inference by significantly compressing the Key-Value (KV) cache into a latent vector. |
| Approach: | They propose a data-efficient fine-tuning method for transitioning from MHA to MLA using a latent vector cache. |
| Outcome: | The proposed architecture reduces the KV cache size of Llama2-7B by 92.19%, with only 1% drop in LongBench performance. |
Copied to clipboard
| Challenge: | Recent work on document-level event argument extraction models each individual event in isolation and therefore causes inconsistency among extracted arguments across events. |
| Approach: | They propose an event-aware argument extraction model with augmented context to improve consistency . they hypothesize that participants tend to play consistent roles across multiple events in a document . |
| Outcome: | The proposed model improves consistency and accuracy of arguments extracted from documents. |
Copied to clipboard
| Challenge: | Experimental results show that the MOS-aware GRM significantly improves fine-grained speech quality discrimination. |
| Approach: | They propose a MOS-aware reward model that incorporates MOS gap into reward function during reinforcement learning. |
| Outcome: | The proposed model significantly improves fine-grained speech quality discrimination. |
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable success across a wide range of tasks, however, they still face challenges in reasoning tasks that require understanding and inferring relationships between distinct pieces of information within text sequences. |
| Approach: | They propose to construct explicit graphs from context and leverage them to enhance LLM reasoning performance on reasoning tasks. |
| Outcome: | Extensive experiments show that the proposed method improves both logical reasoning and multi-hop question answering tasks. |
Copied to clipboard
| Challenge: | Existing research on the allocation of public scarce resources has limitations due to data scarcity and data scariness. |
| Approach: | They propose a framework that integrates Large Language Models into economic simulations . they conduct extensive policy simulation experiments to verify the framework's effectiveness . |
| Outcome: | The proposed framework bridges the gap between theoretical models and real-world dynamics by integrating large language models into economic simulations. |
Copied to clipboard
| Challenge: | Multimodal large language models have demonstrated remarkable performance in visual-language tasks, but their authenticity is often compromised by object hallucinations. |
| Approach: | They propose a multi-frequency perturbation method that leverages both low-frequency and high-frequency features of images to perturb visual feature representations and explicitly suppress redundant frequency-domain features during inference. |
| Outcome: | The proposed method significantly mitigates object hallucinations across various model architectures. |
Copied to clipboard
| Challenge: | Existing methods for multi-turn self-reflection are limited by the Echo Trap problem . the model is limited by its inherent capabilities and repeats earlier reflections to preserve reward signals . |
| Approach: | They propose a tree-structured extension of GRPO for multi-turn self-reflection which enables more accurate advantage estimation. |
| Outcome: | The proposed method mitigates behavior collapse and improves performance across benchmarks. |