Papers by Jiahao Li
Copied to clipboard
| Challenge: | Recent commercial systems such as Suno demonstrate strong capabilities in long-form song generation, but academic research remains non-reproducible due to the lack of publicly available training data. |
| Approach: | They propose a system for long-form song generation with fine-grained style conditioning that includes a licensed synthetic dataset and a song generation model, Muse. |
| Outcome: | The proposed system achieves competitive performance on phoneme error rate, text–music style similarity, and audio aesthetic quality while enabling controllable segment-level generation across different musical structures. |
Copied to clipboard
| Challenge: | Existing GUI agent benchmarks are manually constructed and lack scale and diversity as training environments. |
| Approach: | They propose a GUI agent training system that automatically generates web environments at scale. |
| Outcome: | The proposed system outperforms commercial GUI agents at realistic website construction and improves on OSWorld and Online-Mind2Web. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are hampered by hallucinations, a particularly challenging variant, knowledge overshadowing, which can lead to erroneous outputs even with high-quality training data. |
| Approach: | They propose a framework to analyze and detect knowledge overshadowing by using knowledge circuit analysis to dissect the function of key components in the circuit and how attention pattern dynamics contribute to the phenomenon. |
| Outcome: | Extensive experiments show that the framework can detect and analyze knowledge overshadowing and improves on existing models. |
Copied to clipboard
| Challenge: | Large Language Models suffer from hallucinations, severely undermining their reliability. |
| Approach: | They propose a framework that localizes fact-critical tokens and performs sequential analysis on their hidden states. |
| Outcome: | The proposed framework localizes fact-critical tokens using Factual Criticality . it then performs a focused sequential analysis on their hidden states . |
Copied to clipboard
| Challenge: | Chart2Code is a new benchmark for evaluating the natural language to chart code generation capabilities of large multimodal models. |
| Approach: | They introduce Chart2Code, a new benchmark for evaluating the natural language to chart code generation capabilities of large multimodal models. |
| Outcome: | The proposed benchmark is the first to scale task complexity while capturing diverse scenarios. |
Copied to clipboard
| Challenge: | In-context learning (ICL) is a popular way to stimulate LLM capabilities for downstream tasks due to context length constraints. |
| Approach: | They propose a feature-adaptive and data-scalable in-context learning framework which leverages task-adaptives to promote inference on the downstream task. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on 10 datasets under different data settings and LLM scale. |
Copied to clipboard
| Challenge: | Existing models that use English and local languages have a multilingual gap . a language-informed co-reasoning framework can be used to improve multilingual reasoning . |
| Approach: | They propose a language-informed co-reasoning framework that elicits parallel English and local-language reasoning and abstracts them into structured concepts. |
| Outcome: | Experiments show that Med-CoReasoner improves multilingual reasoning performance by 5% . the framework produces clinically sound and culturally grounded reasoning traces . |
Copied to clipboard
| Challenge: | Existing approaches to transfer a pretrained language model include fine-tuning all the parameters in the language model and adapting all its subsets. |
| Approach: | They propose to select layers based on the variability of their hidden states given a task-specific corpus. |
| Outcome: | The proposed model reduces the computational cost of transfer learning methods without sacrificing performance. |
Copied to clipboard
| Challenge: | Existing models do not analyze human preferences at a finer granularity, which leads to quality issues. |
| Approach: | They propose a set of preference indicators across two major dimensions, text-image consistency and aesthetic quality, and a generative framework to steer the model toward a generation path that more closely aligns with human aesthetic sensibilities. |
| Outcome: | The proposed model improves target recognition accuracy and overall visual aesthetic presentation by focusing on human preferences. |
Copied to clipboard
| Challenge: | Agentic learning increasingly hinges on interaction, yet real-world experience is expensive, limited, and often irreversible at inference time. |
| Approach: | They propose a framework that reframes language modeling as next-state prediction under interaction. |
| Outcome: | The proposed framework evaluates world models in text-based environments . it shows that sufficiently trained models capture coherent environment dynamics . |
Copied to clipboard
| Challenge: | Existing systems with opaque architectures are limiting deep search capabilities for web-augmented large language models. |
| Approach: | They propose a transparent and modular multi-agent framework to democratize deep search for LLMs. |
| Outcome: | The proposed framework outperforms open-source systems in deep reasoning tasks. |
Copied to clipboard
| Challenge: | Existing methods for detecting LLM-generated text rely on statistical features that are insufficient for reliable detection. |
| Approach: | They propose a temperature-sensitive detector that modulates decoding temperature and monitors how probability distributions respond to temperature. |
| Outcome: | The proposed method is based on a temperature sensitivity feature and a simple zero-shot detector built upon normalized temperature sensitivity. |
Copied to clipboard
| Challenge: | Efficient resume parsing is critical for global hiring, yet the lack of dedicated benchmarks for evaluating large language models (LLMs) on multilingual, structure-rich resumes hinders progress. |
| Approach: | They propose to use a human-in-the-loop pipeline to generate 2,500 synthetic resumes spanning 50 templates, 30 career fields, and 5 languages to evaluate large language models. |
| Outcome: | The proposed benchmarks show that the models perform poorly on multilingual resumes and lack of standardized templates. |
Copied to clipboard
| Challenge: | Current mathematical benchmarks focus on evaluating MLLMs’ problem-solving ability, yet there is a crucial gap in addressing more complex scenarios such as error detection. |
| Approach: | They propose to evaluate multimodal error detection by evaluating two sub-tasks error step identification and error categorization. |
| Outcome: | The proposed task evaluates MLLMs' ability to handle multimodal questions compared to text-only models. |
Copied to clipboard
| Challenge: | Recent studies provide the circuit complexity bounds to Transformer-like architectures. position embedding has emerged as a crucial technique in modern large language models. |
| Approach: | They propose to use position embedding to improve Transformer-like architectures by analyzing their circuits and analyzing the results. |
| Outcome: | The proposed model is able to solve canonical tasks without embedding positional information. |
Copied to clipboard
| Challenge: | Existing membership inference attacks require access to complete logits, but such access is often unavailable in real-world deployments where only the generated text is exposed. |
| Approach: | They propose a surrogate-free label-only MIA approach that directly estimates token probabilities through Monte Carlo sampling of the target model. |
| Outcome: | The proposed approach outperforms existing label-only attacks and serves as a foundational density estimator in the label-exclusive setting. |
Copied to clipboard
| Challenge: | Existing methods for in-context learning with large language models focus on using correct or negative examples, ignoring the potential value of incorrect or negative samples. |
| Approach: | They propose a few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. |
| Outcome: | The proposed technique outperforms previous few-shot in-context learning methods on a broad spectrum of related tasks. |
Copied to clipboard
| Challenge: | Existing post-SFT methods for embodied AI are constrained by sparse rewards and action-only optimization, resulting in low sample efficiency, poor consistency, and model degradation. |
| Approach: | They propose to integrate Thought-Centric Preference Optimization (TCPO) into embodied decision-making by transforming sparse reward signals into richer step sample pairs. |
| Outcome: | The proposed approach achieves an average success rate of 26.67% in the ALFWorld environment, and a 6% improvement over RL4VLM. |
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting can mitigate hallucinations by encouraging step-by-step reasoning, but its impact on halluciation detection remains underexplored. |
| Approach: | They conduct an empirical evaluation of CoT prompting in Large Language Models (LLMs) to examine their impact on hallucination detection methods. |
| Outcome: | The proposed method significantly affects the internal states and token probability distributions of the LLM. |
Copied to clipboard
| Challenge: | Existing methods to enhance textual entity prediction neglect the need for external knowledge or encounter high redundancy in the retrieved knowledge. |
| Approach: | They propose a framework that leverages ChatGPT as an implicit knowledge base and heuristically generates auxiliary knowledge for more efficient entity prediction. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on two classic datasets and exhibits a stronger robustness and generalization capability. |
Copied to clipboard
| Challenge: | CPsyExam prioritizes psychological knowledge and case analysis separately, recognizing the significance of applying psychological knowledge to real-world scenarios. |
| Approach: | They propose a psychological benchmark, CPsyExam, constructed from questions from Chinese examination systems. |
| Outcome: | The proposed benchmark prioritizes psychological knowledge and case analysis separately, recognizing the significance of applying psychological knowledge to real-world scenarios. |
Copied to clipboard
| Challenge: | Mamba models demonstrate superior inference efficiency and competitive performance on short-context tasks, but their capacity to comprehend long contexts is limited compared to transformer-based models. |
| Approach: | They propose a model which incorporates selective compression and adaptation techniques within a two-stage re-forward process, incurring minimal additional inference costs overhead. |
| Outcome: | The proposed model improves on the LongBench and L-Eval benchmarks by 3.2 and 1.6 points and attains performance almost on par with same-size transformer models. |
Copied to clipboard
| Challenge: | Existing datasets lack consulting knowledge, resulting in LLMs lacking professional consulting competence. |
| Approach: | They propose a report-based multi-turn dialogue reconstruction framework for Chinese psychological counseling that uses large language models to assist counseling. |
| Outcome: | The proposed framework is open-source and can be used in future research. |
Copied to clipboard
| Challenge: | High-order numerical methods enhance performance in tasks like NLP but introduce a performance-efficiency trade-off due to increased computational overhead. |
| Approach: | They propose an iterative implicit Euler Transformer which simplifies high-order numerical methods by iterating implicit Eule. |
| Outcome: | The proposed method improves accuracy and reduces inference overhead by 55% while retaining 99.4% of the original task accuracy. |
Copied to clipboard
| Challenge: | Existing Trojan attacks require extensive training data and poor generalization, limiting effectiveness and scalability. |
| Approach: | They propose a method for embedding Trojans into plugins using a single edit layer . they find that the method reduces modified parameters by 8-fold and cuts injection time to 25 seconds . |
| Outcome: | The proposed method achieves an average attack success rate of 91%, a 78% improvement over the state-of-the-art (SOTA) method. |
Copied to clipboard
| Challenge: | Experimental results show that representation-based text matching methods suffer from performance degradation due to the lack of interactions between the pair of texts. |
| Approach: | They propose a virtual interaction mechanism that enables deep interaction between texts . they propose 'inteRacTion mechanism' that can be integrated into existing methods as plugins . |
| Outcome: | The proposed method outperforms state-of-the-art models on six text matching benchmarks. |
Copied to clipboard
| Challenge: | Existing methods for multimodal entity linking rely on mention words as retrieval cues, which limits their ability to effectively utilize information from both images and text. |
| Approach: | They propose a visual prompt-guided multimodal entity linking task for a text-image pair . they propose VPWiki to facilitate this task and a framework to capture latent information. |
| Outcome: | The proposed framework outperforms baseline methods on a VPWiki dataset. |
Copied to clipboard
| Challenge: | Parody is an emerging phenomenon on social media, where individuals imitate a role or position opposite to their own . limited available data and deficient diversity in current datasets hinder study of parody . |
| Approach: | They build a dataset of parody users and annotated comments from both English and Chinese corpora to test parody detection and comment sentiment analysis. |
| Outcome: | The proposed datasets provide richer contextual information, which is lacking in existing datasets. |
Copied to clipboard
| Challenge: | Code Language Models learn attention based on statistical input-output token correlations. |
| Approach: | They propose a model-agnostic technique to align CodeLLM attention with human visual attention without architectural changes. |
| Outcome: | The proposed model outperforms baselines in three languages, with gains of over 30 CodeBLEU points in translation and up to 22 BERTScore points in summarization. |
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a key task reliant on textual data. |
| Approach: | They propose a method to transform NER into a multimodal task by using images from the internet as auxiliaries. |
| Outcome: | The proposed method surpasses all text-only baselines and improves F1 score by 1.4% to 2.3% on prominent MNER datasets. |
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have brought significant changes to various domains, especially through autonomous agents. |
| Approach: | They propose a framework that lets agents learn shortcuts from their past tasks and use them for future task execution. |
| Outcome: | The proposed framework enables agents to tackle unseen software-developing tasks more effectively. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have recently advanced the field of Automated Theorem Proving (ATP) Existing cost analyses regulate only the number of sampling passes, ignoring the substantial disparities in sampling costs. |
| Approach: | They propose to integrate two complementary methods into a unified EconRL pipeline to increase pass rates under constrained sampling passes. |
| Outcome: | The proposed method reduces token usage and sample passes while maintaining the original performance. |
Copied to clipboard
| Challenge: | Empirical evaluation shows that MainGEC achieves consistent and significant performance improvements on two benchmark datasets. |
| Approach: | They propose to use mixed-grained weighted training to improve the training effect for GEC by analyzing the inherent discrepancies in annotated training data. |
| Outcome: | Empirical results show that the proposed method achieves significant performance improvements on two benchmark datasets. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit severe hallucinations, which undermine reliability of automated scientific document understanding systems. |
| Approach: | They propose a framework for mitigating scientific measurement hallucinations through enhanced reasoning and targeted optimization. |
| Outcome: | The proposed framework significantly reduces hallucination rates and improves overall accuracy on the MeasEval benchmark. |
Copied to clipboard
| Challenge: | Existing methods for Grounded Multimodal Named Entity Recognition (GMNER) lack a strong correlation between image-text pairs and is ungroundable. |
| Approach: | They propose a framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging large language models as a connecting bridge. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on the existing GMNER dataset and achieves absolute leads of 10.65%, 6.21%, and 8.83% in all three subtasks. |
Copied to clipboard
| Challenge: | Existing methods to improve truthfulness are training-free without modifying the LLM itself. |
| Approach: | They propose a rank-adaptive LoRA method to improve LLM truthfulness that allocates ranks according to truthfulness correlations of LLM modules. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the LLM family and makes the performance of 7B LLMs exceed GPT-4. |
Copied to clipboard
| Challenge: | a study of slow reasoning models for multimodal reasoning finds that they are more prone to fabricating plausible yet false details when confronted with incomplete or misleading visual inputs. |
| Approach: | They conduct the first systematic study of the inverse scaling law in slow-thinking paradigms for multimodal reasoning. |
| Outcome: | The findings suggest that slower reasoning models are more prone to fabricating false details . the study analyzed 5,000-sample hierarchical prompt dataset by 50 participants . |
Copied to clipboard
| Challenge: | Existing methods to extract product attribute value require multiple extractions to obtain all corresponding values. |
| Approach: | They propose an Efficient product Attribute Value Extraction approach using lightweight sparse-layer interaction. |
| Outcome: | The proposed method achieves significant efficiency gains with neutral or marginal loss in performance when the context is long and number of attributes is large. |
Copied to clipboard
| Challenge: | Chinese spelling check (CSC) is a fundamental NLP task that detects and corrects spelling errors in Chinese texts. |
| Approach: | They propose an auxiliary task of Chinese pronunciation prediction to improve CSC . they propose adaptive weighting schemes and a delicate correction strategy . |
| Outcome: | The proposed auxiliary task improves Chinese pronunciation prediction on three benchmarks. |
Copied to clipboard
| Challenge: | Existing jailbreak techniques focus on prompt manipulation or intent obfuscation to bypass safety filters. |
| Approach: | They propose a jailbreak technique that exploits the ability to store, retrieve, and use historical knowledge of Large Language Models (LLMs) they use an inspector agent to extract historical information and an attacker agent to generate adversarial prompts, enabling effective bypassing of safety filters. |
| Outcome: | The proposed jailbreak technique outperforms state-of-the-art jailbreak techniques on six popular models and maintains over 55.4% ASR against defence mechanisms. |
Copied to clipboard
| Challenge: | Existing training methods, such as direct instruction fine-tuning, overlook hierarchical relationships among acceleration patterns. |
| Approach: | They propose a new training paradigm that uses bidirectional tree editing and progressive code acceleration learning to improve LLMs’ CA capabilities. |
| Outcome: | The proposed training paradigm outperforms prompt-enhanced GPT-4 and current training-based methods on average across five programming languages. |
Copied to clipboard
| Challenge: | Existing video LLMs excel at capturing the overall description of a video but lack the ability to demonstrate an understanding of temporal dynamics and localized content within the video. |
| Approach: | They propose a Time-Perception Enhanced Video Grounding via Boundary Perception and Temporal Reasoning to improve LLMs' understanding of video temporality. |
| Outcome: | The proposed method improves on three datasets: ActivityNet, Charades, and DiDeMo (up to 11.2% improvement on R@0.3). |