Papers by Ge Zhang
Copied to clipboard
| Challenge: | if rewards are imperfect, they can adversely affect the alignment of large language models (LLMs). |
| Approach: | They propose a bias-agnostic method to address the issue of reward unfairness from a resource allocation perspective without specifically designing for each type of bias . they apply methods Fairness Regularization and Fairness Coefficient to achieve fairness in rewards. |
| Outcome: | The proposed method achieves fairness in rewards while minimizing biases . it can be applied to verification and reinforcement learning scenarios . |
Copied to clipboard
| Challenge: | DisCo-Speech is a zero-shot controllable text-to-speech framework . standard codecs entangle timbre and prosody, which hinders independent control in continuation-based LMs. |
| Approach: | They propose a disentangled speech codec and an LM-based generator to solve this problem . they propose fusion and reconstruction that merges content and prosody into unified tokens . |
| Outcome: | DisCo-Speech achieves competitive voice cloning and superior zero-shot prosody control. |
Copied to clipboard
| Challenge: | Large language models have demonstrated impressive performance across a wide range of tasks, but this achievement comes with the trade-off of significant computational demands. |
| Approach: | They propose a scaling law that decomposes the overall validation loss and assigns different importance weights to tokens to assess a specific meta-capability. |
| Outcome: | The proposed model can predict the loss trending of models across different levels of computation without a gap between validation loss and model's downstream capabilities. |
Copied to clipboard
| Challenge: | Existing techniques for extending context capabilities in LLMs require additional training procedures and access to datasets with long context (e.g., sequences of 32K tokens). |
| Approach: | They propose a solution to extend context capabilities in Large Language Models by training a single process over a sequence of 4K tokens. |
| Outcome: | The proposed solution significantly reduces the cost of continual-pretraining or fine-tuning over short sequences and improves robustness to diverse relative positions. |
Copied to clipboard
| Challenge: | Recent studies show that learning-based models fail to accurately capture and describe abnormal regions due to data bias. |
| Approach: | They propose a model that compares the current input image with normal images to capture abnormal regions by contrasting the input image and normal images. |
| Outcome: | The proposed model can be easily incorporated into existing models to boost their performance under most metrics. |
Copied to clipboard
| Challenge: | Formality style transfer is a task of automatically transforming text in one particular formality style into another. |
| Approach: | They propose to augment parallel data with three specific data augmentation methods to improve the model's generalization ability and reduce the overfitting risk. |
| Outcome: | The proposed methods significantly improve performance when used to pre-train the model and lead to the state-of-the-art results in the GYAFC benchmark dataset. |
Copied to clipboard
| Challenge: | Current approaches for Multimodal Sentiment Analysis (MSA) rely on parameter-heavy LLMs for classification, overlooking multimodal sentiment reasoning generation in resource-limited environments. |
| Approach: | They propose a multimodal sentiment reasoning distillation model that employs a teacher-assistant-student paradigm to address deployment constraints in resource-limited environments. |
| Outcome: | The proposed model performs well on a resource-limited JMSRC task with only 3B parameters and shows generalization and interpretability. |
Copied to clipboard
| Challenge: | Existing methods do not examine social groups categorised by geographical information, leaving the region-related biases in pre-trained LMs unexplored. |
| Approach: | They propose a hierarchical regional bias evaluation method to quantify regional bias in pre-trained language models. |
| Outcome: | The proposed method evaluates regional bias with regard to comprehensive topics and measures potential regional bias that can be propagated to downstream tasks. |
Copied to clipboard
| Challenge: | Existing methods for large language models (LLMs) are limited by step-by-step decision-making on KGs, or require fine-tuning or pre-training on specific KG. |
| Approach: | They propose a framework that harnesses the global planning abilities of large language models (LLMs) for efficient and accurate KG reasoning. |
| Outcome: | Extensive experiments show that the proposed framework achieves state-of-the-art performance in KGQA tasks, delivering both high efficiency and accuracy. |
Copied to clipboard
| Challenge: | Existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of Large Language Models (LLMs). |
| Approach: | They propose a repository-level benchmark named DevEval to evaluate LLMs' coding abilities in real-world code repositories. |
| Outcome: | The proposed benchmarks show that the LLMs perform better in real-world code repositories than existing benchmarks. |
Copied to clipboard
| Challenge: | Existing studies on data visualization generation from natural languages have not been conducted on Chinese Text-to-Vis. |
| Approach: | They propose to generate a Chinese text-to-vis dataset using a multilingual encoder and a cross-lingual ability. |
| Outcome: | The proposed dataset is challenging and deserves further research. |
Copied to clipboard
| Challenge: | Existing evaluators compress diverse human judgments into a single scalar, leading to brittle alignment and reward hacking. |
| Approach: | They propose a Gaussian-based reinterpretation of reward evaluation as a conditional distribution and a mixture of Gaussians to capture conflicting preference dimensions. |
| Outcome: | The proposed model outperforms scalar baselines in accuracy and generalization. |
Copied to clipboard
| Challenge: | Recent advances in diffusion models have shown impressive performance in many domains, but their ability to follow instructions is still unsatisfactory. |
| Approach: | They propose an algorithm that aligns images to text through iterative image sampling and prompt relabeling with feedback. |
| Outcome: | The proposed algorithm improves on the spatial relation VISOR benchmark by 15.22% compared to previous methods. |
Copied to clipboard
| Challenge: | Medical record reviewers must produce consistent, traceable, guideline-compliant outcomes . longcontext inference is expensive and often degrades as inputs grow . |
| Approach: | a new method compiles textual guidelines into a fixed review tree . a cost-aware split-and-prune search is used to update the tree offline . the algorithm produces consistent, traceable, guideline-compliant outcomes . |
| Outcome: | The proposed system outperforms the strongest non-expert baselines by 84.5–92.8 Macro-F1 . it reduces average I/O volume to 74K input+output characters and average latency to 22s . |
Copied to clipboard
| Challenge: | Empirical studies show that our approach outperforms the SOTA methods in improving the interpretability of text classification models. |
| Approach: | They propose an enhanced variational word masks approach that exploits the Variational Information Bottleneck to obtain task-specific words. |
| Outcome: | Empirical results show that the proposed method outperforms the SOTA methods in improving the interpretability of the model. |
Copied to clipboard
| Challenge: | Existing multimodal large language models lack the ability to memorize, recall, and reason in sustained interactions. |
| Approach: | They propose a multimodal real-world conversation benchmark for evaluating open-ended abilities of multimodal large language models. |
| Outcome: | The proposed benchmarks show that the models perform better in open-ended conversations. |
Copied to clipboard
| Challenge: | Existing language models that use discrete representations for unified processing of various modalities are limited to text generation and do not include multimodal output. |
| Approach: | They propose a multimodal language model that utilizes discrete representations for unified processing of various modalities. |
| Outcome: | The proposed model can be trained stably without any alterations to existing models or training paradigms. |
Copied to clipboard
| Challenge: | Existing LJP models fail to evaluate specific aspects of their performance, such as legal fairness and judicial fairness. |
| Approach: | They propose a suite of functional tests for LJP models to comprehend LJp models’ behaviors and offer diagnostic insights. |
| Outcome: | Extensive tests reveal weaknesses in LJP models and provide diagnostic insights. |
Copied to clipboard
| Challenge: | Hierarchical text classification (HTC) is a challenging task in natural language processing due to its complex taxonomic label hierarchy. |
| Approach: | They propose to use prompts to model hierarchical text classification (HTC) they propose to introduce conditional random fields and Global Pointer to establish hierarchic dependencies . |
| Outcome: | The proposed approach achieves state-of-the-art (SoTA) performance on three public datasets. |
Copied to clipboard
| Challenge: | Existing approaches to few-shot Relation Extraction (RE) are prone to confusion when applying knowledge to a target domain with entirely new types of relations. |
| Approach: | They propose a relation-aware prompt learning method with pre-training to clear confusion by decomposing relation types through an innovative label prompt. |
| Outcome: | The proposed method outperforms previous sota methods and yields better results on cross-domain few-shot RE tasks. |
Copied to clipboard
| Challenge: | Parallel Coordinated Reasoning (PaCoRe) overcomes a central limitation of contemporary language models: their inability to scale test-time compute (TTC) far beyond sequential reasoning under a fixed context window. |
| Approach: | They propose a training-and-inference framework to overcome a central limitation of language models: their inability to scale test-time compute (TTC) under a fixed context window. |
| Outcome: | The proposed model scales to multi-million-token effective TTC without exceeding context limits. |
Copied to clipboard
| Challenge: | Existing Large language models (LLMs) have low pass rates and accuracy on competitive programming tasks. |
| Approach: | They propose a generate-and-edit approach that uses execution results of generated code from LLMs to improve code quality on competitive programming tasks. |
| Outcome: | The proposed method improves pass@1 by 89% on APPS-dev, 31% on apps-test, and 48% on HumanEval over nine popular code generation LLMs with parameter sizes ranging from 110M to 175B. |
Copied to clipboard
| Challenge: | Existing methods for continual learning in language models suffer catastrophic forgetting when learning sequential tasks. |
| Approach: | They propose an orthogonal low-rank adaptation approach for continual learning in language models that uses orthogons to learn sequentially. |
| Outcome: | The proposed approach outperforms state-of-the-art methods on continual learning benchmarks and preserves generalization ability of LLMs on unseen tasks. |
Copied to clipboard
| Challenge: | Large Language Models excel in simple tasks such as generating standalone code units, but real-world software development often involves complex code repositories with complex dependencies and extensive documentation. |
| Approach: | They propose a novel LLM-based agent framework that employs external tools for effective repo-level code generation. |
| Outcome: | The proposed framework outperforms commercial products like Github Copilot in the humanEval benchmark and shows that it is adaptable and efficient across multiple code generation tasks. |
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive capabilities in utilizing external tools, but in practice, they are often exposed to tools that are irrelevant to the user’s query, in which case the desired behavior is to refrain from invocations. |
| Approach: | They propose a new dataset that decouples structural alignment from semantic relevance and propose rebalancing strategies that effectively mitigates structural alignment bias. |
| Outcome: | The proposed approach effectively mitigates structural alignment bias without degrading general tool-use capabilities. |
Copied to clipboard
| Challenge: | Existing models for false information detection on social networks are too harsh for actual social networks that contain both seen and unseen topics simultaneously. |
| Approach: | They propose an open-topic scenario that assumes that all test data topics are seen or unseen by the model, but which is too harsh for actual social networks that contain both seen and unseened topics simultaneously. |
| Outcome: | The proposed model improves on two benchmark datasets and a variety of graph neural networks on two social networks and shows that it is more accurate than existing models. |
Copied to clipboard
| Challenge: | Recent studies have shown that Large Language Models (LLMs) are susceptible to backdoor attacks, where triggers embedded in poisoned data can maliciously alter LLMs’ behaviors. |
| Approach: | They propose to leverage LLMs' generative capabilities to generate human-readable explanations for their decisions, enabling direct comparisons between explanations of clean and poisoned data. |
| Outcome: | The proposed model produces coherent explanations for clean inputs but logically flawed explanations on poisoned data. |
Copied to clipboard
| Challenge: | Existing approaches to cross-domain relation extraction have been limited by domains . data bias between domains can be difficult to fill, especially in few-shot scenarios . |
| Approach: | They propose a framework to bridge the semantic gap caused by data bias between domains . they use syntactic structure, label distribution, and entities to calculate causal effects . |
| Outcome: | The proposed framework fills the domain gap and yields better results on the few-shot task. |
Copied to clipboard
| Challenge: | Existing benchmarks for musical score understanding are narrow in scope, focusing on isolated fragments, short excerpts, or multiple-choice formulations, rather than supporting holistic reasoning over entire scores. |
| Approach: | They propose a benchmark for score-level musical understanding across textual and visual modalities. |
| Outcome: | The musical score understanding benchmark contains 1,800 question-answer pairs from works by Bach, Beethoven, Chopin, Debussy, and others. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are gaining a foothold in Recommender Systems (RS) but there is growing concern that LLMs perpetuate stereotypes and may result in unfair recommendations. |
| Approach: | They propose a counterfactually-fair-prompt method for LLM-based recommendation that is based on unbiased foundation mOdels. |
| Outcome: | The proposed method achieves better recommendation performance with a high level of fairness on two real-world datasets. |
Copied to clipboard
| Challenge: | Recent advances in sequence modeling have highlighted the strengths of the transformer architecture. |
| Approach: | They propose a general lattice transformer for speech translation where the input is the output of the automatic speech recognition (ASR) they propose 'controllable' lattica attention mechanism to consume latent representations. |
| Outcome: | The proposed model outperforms baseline and lattice LSTM on the Chinese-English translation task. |
Copied to clipboard
| Challenge: | Existing explainable recommendation models generate repetitive sentences for different items or empty sentences with insufficient details. |
| Approach: | They propose a visual-enhanced approach to generate rating scores and text explanations using visualization generation and text–image matching discrimination. |
| Outcome: | The proposed approach improves both the text quality and the diversity and explainability of the generated explanations. |
Copied to clipboard
| Challenge: | Training large foundation models for agentic tasks is impractical due to high computational costs, long iteration cycles, and rapid obsolescence as new models are released. |
| Approach: | They propose a method that trains a small LLM to generate supplemental text that helps the larger LLM solve the task more effectively. |
| Outcome: | The proposed approach decouples task-specific optimization from large foundation models . it achieves consistent and significant performance gains across diverse tasks and models - all without gradient access to the actor model. |
Copied to clipboard
| Challenge: | Using machine translation tools for everyday tasks is becoming more commonplace, but a lack of evaluation strategies and alternatives can cause users to over-rely on it. |
| Approach: | They propose to use MT evaluation techniques to promote MT quality and MT literacy among its users. |
| Outcome: | The findings highlight the need for evaluation and NLP explanation techniques to promote MT quality and MT literacy among its users. |
Copied to clipboard
| Challenge: | Reinforcement Learning (RL) is crucial for Video-LLMs with complex spatiotemporal reasoning. |
| Approach: | They propose a framework that decomposes difficulty into two axes in video understanding . they employ efficient, training-free proxies to map data onto a 2D curriculum grid . |
| Outcome: | The proposed framework surpasses strong RL baselines on reasoning and perception tasks. |
Copied to clipboard
| Challenge: | Large Language Models have shown immense potential in multimodal applications, but convergence between textual and musical domains remains unexplored. |
| Approach: | They propose a system that aligns music representations with a frozen LLM . they train the system on an extensive music caption dataset and fine-tune it with instructional data . |
| Outcome: | The proposed system bridges the gap between music audio and textual contexts by combining music captions with a frozen model . it performs well in generating music caption and composing music-related Q&A pairs . the proposed system is available for free download at http://www.musilingo.com/ . |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been widely adopted in real-world dialogue applications, but their robustness is criticized all along. |
| Approach: | They propose to use play-by-play text commentary to build a multi-turn athletic real-world scenario dialogue benchmark to evaluate three critical aspects of multi-turned conversations: ultra multi- turn, interactive multi-twist, and cross-turn tasks. |
| Outcome: | The proposed benchmarks outperform open-source LLMs on three critical aspects of multi-turn conversations: ultra multi-turned, interactive multi- turn, and cross-turn tasks. |
Copied to clipboard
| Challenge: | Entity alignment (EA) aims to identify entities across different knowledge graphs (KGs) that refer to the same real-world object. |
| Approach: | They propose to use large language models to integrate semantic knowledge into EA to identify entities across different knowledge graphs that refer to the same object. |
| Outcome: | The proposed agent outperforms existing methods and achieves state-of-the-art performance on three benchmark datasets. |
Copied to clipboard
| Challenge: | Existing methods for instruction data selection have limitations such as relying on fragile external APIs, being affected by biases in GPT models, or reducing the diversity of the selected instruction dataset. |
| Approach: | They propose an industrial-friendly, expert-aligned and diversity-preserved instruction data selection method: Clustering and Ranking (CaR). |
| Outcome: | The proposed method outperforms Alpaca's existing methods by 32.1% in GPT-4 evaluations. |
Copied to clipboard
| Challenge: | Recent advances in audio diffusion models have significantly improved text-to-audio editing via inversion techniques, but these models typically rely on dense, fixed-step sampling trajectories to maintain structural integrity. |
| Approach: | They propose a model-agnostic Adaptive Trajectory Extrapolation framework that accelerates inversion-based editing process by dynamically evaluating only the most critical generative phases. |
| Outcome: | The proposed framework achieves a 3.9 speedup with negligible loss in fidelity. |
Copied to clipboard
| Challenge: | Existing evaluation methods focus on fluency and factual reliability, while neglecting figurative quality. |
| Approach: | They propose a set of human evaluation metrics focused on the translation of figurative language and a parallel metaphor corpus generated by post-editing. |
| Outcome: | The proposed evaluation protocol estimates four aspects of MT: Metaphorical Equivalence, Emotion, Authenticity, and Quality. |
Copied to clipboard
| Challenge: | Existing LLMs are constrained by their pre-trained context lengths, leading to performance issues . elucidating this limitation, we propose a training-free solution to the context length limitation in LLM applications . |
| Approach: | They propose a method that integrates hierarchical rotary position embedding into LLMs without extra training costs. |
| Outcome: | The proposed method improves performance on language modeling and long code completion tasks. |
Copied to clipboard
| Challenge: | Larger models or those trained on fewer tokens exhibit less quantization-induced degradation (QiD), while smaller, well-trained models face significant performance losses. |
| Approach: | They propose to use QiD to measure an LLM’s training levels and determine the number of training tokens required for fully training LLMs of various sizes. |
| Outcome: | The proposed scaling laws can predict the quantization performance of different-sized LLMs trained with tokens. |
Copied to clipboard
| Challenge: | Existing models for emotion understanding do not capture fundamental features of synthesized speech. |
| Approach: | They evaluate emotion recognition models on synthesized speech using SER models and generative models. |
| Outcome: | The proposed model can't generalize to synthesized speech because of speech token prediction . generative models tend to infer emotion from textual semantics while ignoring paralinguistic cues. |
Copied to clipboard
| Challenge: | Current music information retrieval systems struggle to meet linguistic diversity challenges . current systems struggle with text queries in non-English languages . |
| Approach: | They propose a music information retrieval system that supports both ABC notation and MIDI . CLaMP 2 includes a multilingual text encoder and a multiple-modal music encoder . |
| Outcome: | The proposed system achieves state-of-the-art results in multilingual semantic search and music classification across modalities. |
Copied to clipboard
| Challenge: | a recent study shows that large language models have limited generalization in low-resource languages like Chinese. |
| Approach: | They propose to evaluate the zero-shot generalizability of large language models to the Chinese language . they release only half of the dataset publicly, with the remainder kept private . |
| Outcome: | The Chinese Instruction-Following Benchmark evaluates the generalizability of LLMs to the Chinese language. |
Copied to clipboard
| Challenge: | Temporal reasoning in multi-session dialogues presents a significant challenge which has been under-studied in previous temporal reasoning benchmarks. |
| Approach: | They propose to augment LoCoMo dialogues and create multi-choice QAs to construct a temporal reasoning evaluation task and a framework to enhance temporal thinking capabilities of LLM-agents. |
| Outcome: | The proposed framework significantly improves temporal reasoning performance compared to baseline methods, raising from 29.83 on GPT-4o via standard prompting to 77.67 via the proposed framework. |
Copied to clipboard
| Challenge: | Existing methods to improve the efficiency of GEC are not efficient enough for GEC. |
| Approach: | They propose a language-independent approach to improve the efficiency of GEC by dividing the task into two subtasks: ESD and ESC. |
| Outcome: | The proposed approach performs comparably to conventional seq2seq approaches in English and Chinese GEC benchmarks with less than 50% time cost for inference. |
Copied to clipboard
| Challenge: | a method for user targeting is developed to identify online users to whom an ad should be targeted. |
| Approach: | They propose a method for automatic augmentation of positive and negative clickthrough data for user targeting models. |
| Outcome: | The proposed method can increase positive and negative instances of positive training instances on two datasets. |
Copied to clipboard
| Challenge: | Current outcome-centric verification paradigms neglect potential errors in the derivation process. |
| Approach: | They propose a process-aware RLVR training paradigm utilizing verifiers selected via **PRIME**. |
| Outcome: | The proposed approach outperforms the baseline verification paradigm on AIME24, AIME25, and Beyond-AIME models. |
Copied to clipboard
| Challenge: | Existing video metrics are lagging behind in providing reliable scores over generated videos due to lack of large-scale human-annotated dataset. |
| Approach: | They propose to use VideoFeedback to train a human-annotated multi-aspect score over 37.6K synthesized videos from 11 existing video generative models. |
| Outcome: | The proposed model outperforms the prior best metrics by 50 points in the test. |
Copied to clipboard
| Challenge: | Chinese Spelling Correction (CSC) lacks large-scale high-quality corpora due to labor-intensive labeling of spelling errors in real-life writing or typing scenarios. |
| Approach: | They propose to use OCR/ASR-based generation to refine Chinese Spelling Correction models on random replacement-based corpora and filter them based on prediction confidence. |
| Outcome: | The proposed model outperforms existing models on three widely-used benchmarks while significantly alleviating over-correction. |
Copied to clipboard
| Challenge: | Existing approaches to RLVR use multiple-choice questions as verifiable rewards . however, not all tasks provide reliable verification . |
| Approach: | They propose a framework that actively constructs high-quality distractors to block elimination shortcuts and promote deep reasoning. |
| Outcome: | The proposed method significantly improves reasoning capabilities of Large Language Models. |
Copied to clipboard
| Challenge: | Existing benchmarks primarily focus on Python and are limited in terms of language diversity. |
| Approach: | They propose a multilingual debugging benchmark that includes 3.9K test samples of 20 programming languages and introduces the debug instruction corpora MdEval-Instruct by injecting bugs into the correct multilingual queries and solutions. |
| Outcome: | The proposed benchmark includes 3.9K test samples of 20 programming languages and covers the automated program repair task, bug localization task, and bug identification task. |
Copied to clipboard
| Challenge: | Existing methods combine quantization with parameter-efficient fine-tuning but fail to meet practical performance requirements. |
| Approach: | They propose a measure and moment approach to optimize objective function for superior fine-tuning results by scaling the update process through a gradient. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on tasks like text generation, summarization, and understanding. |
Copied to clipboard
| Challenge: | Existing methods for surfacing symbolic reasoning capabilities are limited to narrow tasks . arithmetic computations are unnatural to perform in pure language space, and hence present difficulties for LLMs. |
| Approach: | They propose a natural language embedded program framework for solving symbolic reasoning tasks. |
| Outcome: | The proposed framework improves on strong baselines across math and symbolic reasoning, text classification, question answering, and instruction following tasks. |
Copied to clipboard
| Challenge: | Existing models that mess up or drop the core structural information of input graphs are lacking in graph-to-text generation. |
| Approach: | They propose to leverage richer training signals to guide a graph-to-text generation model by focusing on autoencoding losses and back-propagating the losses to better calibrate the model. |
| Outcome: | Experiments on two benchmarks show the proposed model over a state-of-the-art model . two types of autoencoding losses are used to back-propagate the model based on multitask training . |
Copied to clipboard
| Challenge: | Currently, long-context language models are limited by the lack of a rigorous evaluation framework for long code understanding. |
| Approach: | They propose to use a long code understanding benchmark LongCodeU to evaluate LCLMs' long code comprehension ability for practical applications. |
| Outcome: | The proposed benchmarks show that current LCLMs are limited in their long code understanding ability, particularly when the long code length is greater than 32K, falling far short of their claimed 128K to 1M context windows. |
Copied to clipboard
| Challenge: | Existing approaches to formalizing mathematical statements face limitations in accuracy, especially in the context of complex, highlevel problems that involve sophisticated mathematical reasoning. |
| Approach: | They propose a CriticLean framework that elevates the role of the critic from a passive validator to an active learning component and introduce a benchmark to measure models’ ability to distinguish semantically correct from incorrect formalizations. |
| Outcome: | The proposed framework outperforms open- and closed-source benchmarks and shows that it significantly outperformed existing models. |
Copied to clipboard
| Challenge: | Despite LLMs' impressive capabilities in musical knowledge, music reasoning remains an unsolved task. |
| Approach: | They propose an open-source large language model (LLM) that integrates intrinsic musical abilities into LLaMA2 and GPT-3.5. |
| Outcome: | The proposed model can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers. |
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have led to progress in tackling complex reasoning tasks that combine textual and visual information. |
| Approach: | They introduce a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. |
| Outcome: | The proposed model performs lower on MMMU-Pro than on the previous benchmark, ranging from 16.8% to 26.9%. |
Copied to clipboard
| Challenge: | Recent advances in deep learning have led to significant improvement of document-level neural machine translation (NMT). |
| Approach: | They propose a long-short term masking self-attention on top of the standard transformer to capture the long-range dependence and reduce the propagation of errors. |
| Outcome: | The proposed model captures the long-range dependence and reduces errors on two publicly available document-level datasets. |
Copied to clipboard
| Challenge: | MLLMs that use domain-specific data are limited in understanding cultural heritage artifacts such as ancient Greek pottery . supervised fine-tuning improves adaptation to domain knowledge, but it struggles with deeper reasoning tasks. |
| Approach: | They propose a visual question-answer tool that augments SFT with reinforcement learning using verifiable rewards. |
| Outcome: | The proposed model outperforms baseline models on reasoning-intensive questions on ancient Greek pottery. |
Copied to clipboard
| Challenge: | Existing defense approaches focus on developing new model structures or training algorithms, but they do little to tap the potential of training instances. |
| Approach: | They propose a method that can distinguish between robust and non-robust instances according to the model’s sensitivity to perturbations on individual instances during training. |
| Outcome: | The proposed method can distinguish between robust and non-robust instances according to the model’s sensitivity to perturbations on individual instances during training. |
Copied to clipboard
| Challenge: | TrickCatcher generates test cases that pass existing tests yet contain bugs . a recent study found that tricky bugs are not detected by test suites . |
| Approach: | They propose an LLM-powered approach to generating test cases for uncovering bugs in plausible programs . they use a PUT and specification to generate program variants, an input generator and an Llm to construct test inputs . |
| Outcome: | The proposed approach achieves recall, precision, and F1 scores that are 1.80, 2.65, and 1.66 . trickCatcher generates program variants based on the program under test and its specification . |
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks. |
| Approach: | They propose a framework that optimizes MAS prompts as a maximum a posteriori problem and then iteratively updates agent prompts. |
| Outcome: | The proposed framework surpasses manual and automated benchmarks in multiple tasks and provides general guidelines for building more reliable and principled multi-agent systems in the future. |
Copied to clipboard
| Challenge: | Summarization is an important application of Large Language Models. |
| Approach: | They integrate human-annotated and model-generated natural language explanations to elucidate how a summary deviates and becomes inconsistent with its source article. |
| Outcome: | The proposed model provides rationales for its judgments and improves its accuracy significantly. |
Copied to clipboard
| Challenge: | ReCaLL (Relative Conditional Log-Likelihood) is a membership inference attack that can detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities. |
| Approach: | They propose a membership inference attack to detect LLMs’ pretraining data by leveraging their conditional language modeling capabilities. |
| Outcome: | The proposed model achieves state-of-the-art performance on the WikiMIA dataset, even with random and synthetic prefixes, and can be further improved using an ensemble approach. |
Copied to clipboard
| Challenge: | Existing evaluation methodologies for multimodal large language models are limited in evaluating objective queries without considering real-world user experiences. |
| Approach: | They propose to evaluate multimodal large language models with per-sample criteria using potent MLLM as the judge. |
| Outcome: | The proposed evaluation paradigm shows that it can be used to evaluate multimodal large language models with per-sample criteria. |
Copied to clipboard
| Challenge: | Existing approaches ignore relationships between medical items and statuses in the multi-turn doctor-patient dialogue. |
| Approach: | They propose a task to extract structured medical information from free text dialogues . they propose 'Dialogue Medical Information Extraction' to model relationships between items . |
| Outcome: | The proposed model outperforms previous models and achieves state-of-the-art performance on the public benchmark data set. |
Copied to clipboard
| Challenge: | MLLMs perform poorly on traditional culture images, indicating limitations in understanding high-level semantics and lacking a deep knowledge base of Chinese traditional culture. |
| Approach: | They propose to use Chinese images to assess MLLMs' higher-order perception and understanding of Chinese visual content. |
| Outcome: | The proposed model incorporates images that represent Chinese traditional culture, such as famous Chinese traditional paintings, to ensure the authenticity of the Chinese context. |
Copied to clipboard
| Challenge: | Large language models are increasingly used in decision support systems and workflows . traditional computational paradigms for decision-making under uncertainty choose an option that maximizes expected utility or payoff . |
| Approach: | They compare large language models as decision support systems and agentic workflows . they find that LLMs cluster into reasoning models and conversational models . |
| Outcome: | The proposed models differ in their ability to perform tasks and their ability in a human-like way. |
Copied to clipboard
| Challenge: | Metaphors are a prominent linguistic device in human language and literature, as they add color, imagery, and emphasis to enhance effective communication. |
| Approach: | They propose a large-scale high quality annotated Chinese Metaphor Corpus . they use a set of guidelines to ensure the accuracy and consistency of their annotations . |
| Outcome: | The proposed corpus generates metaphors that resonate more with real-world intuition. |
Copied to clipboard
| Challenge: | Recent Large Multimodal Models (LMMs) have shown promising potential for performing end-to-end KIE directly from document images. |
| Approach: | They propose a benchmark to evaluate the performance of Large Multimodal Models (LMMs) using a constrained-category KIE track and an open-categorical KIE Track. |
| Outcome: | Experiments on 15 state-of-the-art LMMs show performance degradation under diverse schema definitions, long-tail key fields, and complex layouts, along with pronounced performance disparities across different document types and scenarios. |
Copied to clipboard
| Challenge: | Current code generation models produce errors concentrated at specific error-prone points, affecting accuracy of code. |
| Approach: | They propose a framework that focuses preference optimization on error-prone areas . focused-DPO improves the accuracy and reliability of code generation by reducing common errors . |
| Outcome: | The proposed framework improves code generation by focusing on error-prone areas. |
Copied to clipboard
| Challenge: | a large-scale empirical evaluation of hallucination detection metrics is conducted . hallucinosity is a significant obstacle to the reliability and widespread adoption of language models . |
| Approach: | They conduct large-scale empirical evaluation of hallucination detection metrics . they compare hallucinian language models, language models and decoding methods . |
| Outcome: | The results show that the evaluations of hallucination detection metrics fail to align with human judgments, they say . they also show that evaluations with LLM-based evaluation yield the best overall results . |
Copied to clipboard
| Challenge: | LegalBench evaluated 20 LLMs in 162 legal tasks in 20 countries and jurisdictions. |
| Approach: | They present a comprehensive evaluation of 21 popular Large Language Models and the first comparative analysis of the empirical results. |
| Outcome: | The proposed benchmarks are based on the Bloom’s cognitive taxonomy and are compared to 21 popular LLMs. |
Copied to clipboard
| Challenge: | Existing models lack multimodal understanding capabilities, resulting in closed-source model that does not support multimodal interleaved sequences. |
| Approach: | They propose a foundation model built on multimodal tokens capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. |
| Outcome: | The proposed model is able to understand speech, text, images, and videos in an end-to-end, autoregressive manner. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly vulnerable to elusive and implicit intentions, causing security risks and compromising user experience. |
| Approach: | They propose a method to detect and mitigate implicit jailbreak attacks using LLMs by unearthing real intentions and a greedy gradient-based algorithm to remove the least important parts of a sentence. |
| Outcome: | The proposed method reduces attacks success rate and Harmful Score while maintaining overall model performance. |
Copied to clipboard
| Challenge: | Sequence-to-sequence (seq2sequ) models have a weakness: they cannot always generate sentences without grammatical errors. |
| Approach: | They propose to use automatic grammatical error correction to improve seq2seq models . they conduct experiments on machine translation, formality style transfer, sentence compression and simplification . |
| Outcome: | The proposed system can improve grammaticality of generated text and improve formal style tasks. |
Copied to clipboard
| Challenge: | Existing image captioning systems generate narrative captions for images, which are spatially detached from the image in presentation. |
| Approach: | They propose a task called captioning on image which generatesense captions at different locations of the image based on contextual information. |
| Outcome: | The proposed model achieves the best results in both captioning accuracy and diversity aspects. |
Copied to clipboard
| Challenge: | Existing methods for idiomatic expression generation lack parallel data and manual annotations. |
| Approach: | They propose an iterative LLM-SLM collaborative framework that replaces human supervision for idiomatic expression data generation. |
| Outcome: | The proposed framework outperforms DeepSeek-R1 in Chinese Idiom Polishing with a 25.2% improvement in accuracy. |
Copied to clipboard
| Challenge: | Existing multilingual table benchmarks suffer from geolinguistic imbalance - overrepresenting certain languages and lacking sufficient scale for rigorous cross-lingual analysis. |
| Approach: | They propose a framework for massively multilingual table question answering that includes tables expanded to 97 languages from Chinese and English sources. |
| Outcome: | Experiments on state-of-the-art LLMs show that synthetically generated training data significantly boosts performance, especially for low-resource languages. |
Copied to clipboard
| Challenge: | Existing methods for evaluation of large language models are inefficient and inefficient due to inaccuracy of standard metrics in human perception of text quality and inefficiency in sampling informative test examples. |
| Approach: | They propose a sample-efficient human evaluation method for large language models based on the principle of MAximum Discrepancy (MAD) competition. |
| Outcome: | The proposed method achieves the “golden” ranking of LLMs with a minimum set of input instructions, which in turn reveal their relative strengths and weaknesses. |
Copied to clipboard
| Challenge: | Existing Chinese preference datasets suffer from limited scale, restricted domain coverage, and insufficiently rigorous data validation. |
| Approach: | They propose an LLM-based data annotation pipeline with no human intervention to annotate Chinese preference datasets. |
| Outcome: | The proposed pipeline outperforms existing Chinese preference datasets on AlignBench and Chinese Reward Benchmark. |
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) relies on complex methodologies like Proximal Policy Optimization (PPO) that require extensive hyper-parameter tuning and pose challenges in sample efficiency and stability. |
| Approach: | They propose an innovative framework that leverages direct preference optimization techniques but extends them by estimating the conditionally optimal policy directly from the model’s responses. |
| Outcome: | The proposed framework matches and exceeds the effectiveness of Proximal Policy Optimization (PPO) in terms of convergence speed and alignment of model responses with human preferences. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant advances in code generation through the ‘Chain-of-Thought’ prompting technique. |
| Approach: | They propose a framework which aims to transfer LLMs’ reasoning capabilities to smaller models through distillation. |
| Outcome: | The proposed framework improves the smaller model's code generation performance by over 130% on the APPS benchmark. |
Copied to clipboard
| Challenge: | Existing methods for group relative policy optimization suffer from entropy collapse . Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain stability. |
| Approach: | They propose a framework that provides targeted mechanisms for exploration and stabilization. |
| Outcome: | The proposed framework expands search space on difficult prompts while preventing entropy growth uncontrollably. |
Copied to clipboard
| Challenge: | Multi-modal information retrieval (MMIR) is a rapidly evolving field . current benchmarks for image-text pairings overlook the scientific domain . |
| Approach: | They develop a scientific domain-specific MMIR benchmark to evaluate image-text pairings using open-access research paper corpora. |
| Outcome: | The proposed benchmarks are based on 530K image-text pairs extracted from scientific documents with detailed captions. |
Copied to clipboard
| Challenge: | Existing Vision-Language Models (VLMs) lack spatial reasoning, despite text-based CoTs . e-ViC reframes spatial intelligence as a verifiable, tool-using capability, argues a new study. |
| Approach: | They propose a framework that moves reasoning beyond text into the visual domain . they ground reasoning in pixel-level interactions to enable human-like "look-and-confirm" strategies . |
| Outcome: | The proposed framework outperforms existing Vision-Language Models with an average gain of 10.1%. |
Copied to clipboard
| Challenge: | Discourse dependency parsing is a task that requires a large amount of training data, but there is little research on it. |
| Approach: | They propose to adapt unsupervised syntactic dependency parsing methods for unsupervised discourse dependency parses using unlabeled training data. |
| Outcome: | The proposed methods outperform existing methods in semi-supervised and supervised settings and outperformed existing methods. |
Copied to clipboard
| Challenge: | Existing methods for learning human values do not consider contextual and abstract nature of human values. |
| Approach: | They propose a reinforcement learning based method that embeds human values judgements into each step of language generation. |
| Outcome: | The proposed method improves on human values judgements and shows higher alignment performance. |
Copied to clipboard
| Challenge: | Strategic reasoning requires Large Language Model (LLM) agents to adapt their strategies dynamically in multi-agent environments. |
| Approach: | They propose a framework that enables Large Language Models to achieve varying levels of strategic depth by recursive mechanisms that allow agents to form higher order beliefs about others' beliefs. |
| Outcome: | The proposed framework enables LLMs to achieve varying levels of strategic depth, allowing agents to form higher order beliefs—beliefs about others’ beliefs. |
Copied to clipboard
| Challenge: | Code LLMs lack reproducible data pipelines and training protocols for reproducible advancements in code intelligence. |
| Approach: | They propose a top-tier code LLM that releases model weights and inference code . reproducible data pipelines, rigorous experimental ablation results and training protocols are included . |
| Outcome: | The proposed model achieves comparable performance to leading models and serves as an "open cookbook" reproducible training data, rigorous experimental ablation results, and detailed training protocols are also included in the model. |
Copied to clipboard
| Challenge: | Existing domain-specific code benchmarks focus on assessing what knowledge LLMs possess rather than how they acquire and apply new knowledge. |
| Approach: | They propose a benchmark to evaluate domain specialization methods in real-world software development. |
| Outcome: | KOCO-bench is a new benchmark for evaluating domain specialization methods in real-world software development. |
Copied to clipboard
| Challenge: | InternLM-Law is a large language model (LLM) tailored for addressing diverse legal tasks related to Chinese laws. |
| Approach: | They introduce a large language model (LLM) tailored for addressing diverse legal tasks related to Chinese laws. |
| Outcome: | The proposed model performs better than existing models in a variety of legal tasks related to Chinese laws. |
Copied to clipboard
| Challenge: | Existing methods to extract unseen relations require laborious manual annotation . a new approach uses fine-grained matching to reduce manual annotation cost . |
| Approach: | They propose an efficient multi-grained matching approach that uses virtual entity matching to reduce manual annotation cost. |
| Outcome: | The proposed approach outperforms the state-of-the-art methods and achieves inference efficiency and accuracy in zero-shot relation extraction tasks. |
Copied to clipboard
| Challenge: | Unlike image or text classification, speech classification tasks are particularly challenging due to the difficulty in capturing the acoustic, semantic, and contextual representations. |
| Approach: | They propose a dataset pruning method that coarsely filters redundant samples using DBSCAN clustering on Mel-Frequency Cepstral Coefficients (MFCC) features. |
| Outcome: | The proposed method achieves 49.5% improvement in WA on the MEAD dataset and 41.9% reduction in EER on speaker identification tasks. |
Copied to clipboard
| Challenge: | Current multimodal benchmarks focus on facts within individual images, but neglect associative relations among multiple images. |
| Approach: | They propose a multi-image relational association task and a MMRA benchmark to evaluate LVLMs. |
| Outcome: | The proposed benchmarks show that entity-level multi-image perception tasks pose greater challenges than image-level tasks. |
Copied to clipboard
| Challenge: | Existing studies focus on *broadening* the training set with data augmentation techniques to maximize such benefits. |
| Approach: | They propose a method that embeds problem reflection into each training instance. |
| Outcome: | The proposed method enhances performance in standard and complex scenarios that require reflective thinking. |
Copied to clipboard
| Challenge: | a recent study shows that agent research practices are far from standard, rigorous . lack of a standard evaluation protocol makes previous works not reproducible, authors say . |
| Approach: | They conduct an empirical study on the GAIA benchmark to investigate agent design choices . they find that lack of a standard evaluation protocol makes previous works not reproducible . |
| Outcome: | The proposed framework achieves state-of-the-art performance among open-source projects. |
Copied to clipboard
| Challenge: | Existing methods to extract genre-specific and genre-agnostic features require great human effort. |
| Approach: | They propose to use two encoders to explicitly extract genre-specific and genre-agnostic features. |
| Outcome: | The proposed approach outperforms the state-of-the-art by 1.7% on three distinct genres. |
Copied to clipboard
| Challenge: | ConceptMath evaluates concept-wise mathematical reasoning of Large Language Models (LLMs) Existing benchmarks that evaluate general mathematical reasoning with an average accuracy fail to probe the fine-grained failure modes of mathematical reasoning on specific datasets. |
| Approach: | They introduce a bilingual, fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models. |
| Outcome: | The proposed benchmarks evaluate concept-wise mathematical reasoning of Large Language Models with concept-based accuracies. |
Copied to clipboard
| Challenge: | Contextualized embeddings are expensive and resource-demanding, hence environmentally unfriendly. |
| Approach: | They propose a method to convert contextualized embeddings from pre-trained models into static embeddables using synonym knowledge and weighted vector distribution. |
| Outcome: | The proposed method outperforms baseline embeddings by a large margin through extrinsic and intrinsic tasks. |
Copied to clipboard
| Challenge: | Recent deep generative models have already shown encouraging * Equal contribution. |
| Approach: | They propose to use generic instruction-tuned LLMs as direct text-to-sequence generators to achieve this goal. |
| Outcome: | Recent studies show that reflection improves sequence quality and alignment while maintaining competitive foldability. |
Copied to clipboard
| Challenge: | Experimental results confirm the substantial superiority of GranuSum on multi-granularity summarization over strong baselines. |
| Approach: | They propose to rank events by their salience and annotate a benchmark for GranuSum that contains multiple summaries at different granularities for each document cluster. |
| Outcome: | The proposed framework is capable of producing multi-granular summaries in unsupervised manner over strong baselines. |
Copied to clipboard
| Challenge: | Existing datasets for Chinese instruction tuning are not well-aligned with Chinese users’ interaction patterns. |
| Approach: | They propose to use Chinese instruction tuning datasets to improve instruction fine-tuning for Chinese users. |
| Outcome: | The proposed dataset shows that Chinese models achieve competitive performance in diverse benchmarks. |
Copied to clipboard
| Challenge: | Existing training methods for code generation do not improve code correctness and efficiency. |
| Approach: | They propose a framework that integrates preference learning into code generation to improve code correctness and efficiency. |
| Outcome: | The proposed framework improves code correctness and efficiency by integrating preference learning into code generation. |
Copied to clipboard
| Challenge: | Existing repository-level code completion benchmarks focus on a limited number of languages . existing benchmarks report overall average scores of different languages ignoring fine-grained abilities . |
| Approach: | They propose to use repository-level code completion benchmarks to evaluate general code intelligence abilities across languages for existing code Large Language Models. |
| Outcome: | The proposed benchmarks improve the code completion abilities of existing LLMs by using two types of annotations on the parsed syntax tree. |
Copied to clipboard
| Challenge: | Existing evaluations emphasize final accuracy or coarse token counts, and lack automated tools to separate essential logic from structural redundancy. |
| Approach: | They propose a graph-driven framework that quantifies reasoning efficiency by converting free-form CoTs into directed dependency graphs and extracting the Shortest Effective Path needed to reach a correct solution. |
| Outcome: | Evaluating 21 LRMs, the proposed framework quantifies reasoning efficiency by converting free-form CoTs into directed dependency graphs and extracting the Shortest Effective Path (SEP) needed to reach a correct solution. |
Copied to clipboard
| Challenge: | In order to improve translation efficiency, human translators perform post-editing on machine translations to correct errors. |
| Approach: | They propose to use the parameterized objective function of neural machine translation to deal with the TS problem without additional training. |
| Outcome: | The proposed method improves translation quality by 10.6 BLEU and reduces time overhead by 63.4% on benchmark datasets. |
Copied to clipboard
| Challenge: | Existing MLLM benchmarks and unified evaluation frameworks cannot accurately and efficiently reflect the ability of MLMLs. |
| Approach: | They propose a semi-automated benchmark curated using a pipeline that filters out uninformative samples and eliminates answer leakage by focusing on tasks that require image-based understanding. |
| Outcome: | The proposed benchmark reduces the number of samples by 76% and evaluation time by 77% while it can more effectively distinguish different models’ abilities. |
Copied to clipboard
| Challenge: | Existing approaches to optimize sequential recommendation systems rely on item ID sequences, but they lack collaborative knowledge and limited controllability. |
| Approach: | They propose a simple bi-tuning framework with collaborative information for controllable Large Language Model-based Sequential Recommendation (Laser) they incorporate learnable virtual tokens at prefix and suffix of input text to adapt LLMs with collaborative knowledge . |
| Outcome: | The proposed framework outperforms state-of-the-art recommendations on real-world datasets. |
Copied to clipboard
| Challenge: | Recent studies have demonstrated that masked diffusion models (MDMs) can surpass autoregressive models (ARMs) in various tasks. |
| Approach: | They propose a method to calibrate early token predictions without demonstration data by distilling an unnormalized target distribution into the original model. |
| Outcome: | Experiments on math, planning, and RLHF tasks show that COPSD improves both effectiveness and efficiency, and further enhances performance when combined with supervised fine-tuning. |
Copied to clipboard
| Challenge: | Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs). |
| Approach: | They propose a hybrid-policy optimization approach that synergizes internal exploitation with external data to achieve stronger reasoning capabilities. |
| Outcome: | The proposed approach achieves state-of-the-art performance on six math reasoning benchmarks and superior performance on out-of distribution reasoning tasks. |
Copied to clipboard
| Challenge: | Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries. |
| Approach: | They propose to conduct efficient text-video Retrieval with a salient-and-correlated AdaPter . they propose a low-rank modulation module to refine per-image features from frozen CLIP backbone . |
| Outcome: | Experiments on four TVR datasets show that the proposed method performs better than other methods. |
Copied to clipboard
| Challenge: | Existing approaches to offline reinforcement learning (RL) focus on learning value functions or policy gradients, but they view it as a sequence modeling task. |
| Approach: | They propose a method that integrates multimodal and pre-trained language models to transform offline reinforcement learning into a supervised learning task by integrating state information derived from images and action-related data obtained from text. |
| Outcome: | The proposed approach outperforms baselines on Atari and OpenAI Gym environments while promoting long-term strategic thinking. |
Copied to clipboard
| Challenge: | Modern AI agents rely on Large Language Models (LLMs) as their reasoning engines, but they still face the challenge of generating meaningful reflections due to inadequate error analysis and a reliance on rare successful trajectories. |
| Approach: | They propose a framework for self-learning agents powered by a retrospective language model that generates reflections during inference. |
| Outcome: | The proposed framework outperforms reflection-based baselines on three challenging benchmarks. |
Copied to clipboard
| Challenge: | Recent advances in o1-like models have generated long Chain-of-Thought reasoning steps to improve the reasoning abilities of existing Large Language Models (LLMs). |
| Approach: | They propose a DeltaBench to analyze the quality and effectiveness of o1-like models and measure their ability to detect errors in long COT reasoning. |
| Outcome: | The proposed model can detect errors in long COT reasoning. |
Copied to clipboard
| Challenge: | OpenCodeInterpreter-33B provides a high level of performance for code generation, executing, and iterative refinement. |
| Approach: | They propose a family of open-source code systems for generating, executing, and iteratively refining code. |
| Outcome: | The OpenCodeInterpreter-33B performs well on humanEval, MBPP, and EvalPlus benchmarks. |
Copied to clipboard
| Challenge: | Recent studies have highlighted a tendency among large language models to refuse to answer benign queries. |
| Approach: | They propose a model-agnostic approach to reduce excessive attention to harmful words like ‘kill’ and a method to decode the next-token predictions by contrastive decoding. |
| Outcome: | The proposed approach reduces the refusal rate by 20% while having little impact on safety. |