Papers by Kai Zheng

37 papers
Bridge Video and Text with Cascade Syntactic Structure (C18-1)

Copied to clipboard

Challenge: Using LSTM-CSS, we construct basic syntactic structure by completing syntastic structure.
Approach: They propose a video captioning approach that progressively completes syntactic structure by a conditional random field to construct basic syntaktic structure.
Outcome: The proposed method produces natural sentences with 42.3% and 28.5% accuracy compared to state-of-the-art methods.
FastMCTS: A Simple Sampling Strategy for Data Synthesis (2025.acl-long)

Copied to clipboard

Challenge: Existing methods for generating multi-step reasoning data rely on rejection sampling, which generates trajectories independently and suffers from inefficiency and imbalanced sampling across problems of varying difficulty levels.
Approach: They propose a data synthesis strategy inspired by Monte Carlo Tree Search . it offers step-level evaluation signals and promotes balanced sampling .
Outcome: Experiments show that FastMCTS generates 30% more correct reasoning paths than rejection sampling.
ReCo: Reliable Causal Chain Reasoning via Structural Causal Recurrent Neural Networks (2022.emnlp-main)

Copied to clipboard

Challenge: Causal chain reasoning models suffer from two main transitive problems: threshold effect and scene drift.
Approach: They propose a framework that uses exogenous variables to represent causal pairs and estimates the threshold and scene contradictions using structural causal recurrent neural networks.
Outcome: The proposed framework outperforms baselines on Chinese and English CCR datasets.
AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning (2024.emnlp-main)

Copied to clipboard

Challenge: Recent advances in memory-efficient zeroth-order methods have limited their widespread adoption due to performance drops and a high risk of divergence.
Approach: They propose a memory-efficient zeroth-order framework to improve performance and convergence of the MeZO methods by using only forward passes.
Outcome: The proposed framework improves performance and convergence of the proposed methods on Roberta-Large and Llama-2-7B models.
NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models (2023.acl-long)

Copied to clipboard

Challenge: Existing backdoor attacks against prompt-based learning involve injecting back doors into embedding layers or word embedders.
Approach: They propose a backdoor attack against prompt-based learning that injects backdoors into embedding layers or word embeddable vectors.
Outcome: The proposed backdoor attack outperforms two state-of-the-art models on six NLP tasks and three prompting strategies.
RubricBench: Aligning Model-Generated Rubrics with Human Standards (2026.acl-long)

Copied to clipboard

Challenge: Existing benchmarks lack discriminative complexity and ground-truth rubric annotations required for rigorous evaluation.
Approach: They propose a curated benchmark with 1,147 pairwise comparisons to assess the reliability of rubric-based evaluation.
Outcome: The proposed benchmarks show that they support diverse domains, exhibit discriminative ability, provide high-quality annotations, and include human-authored rubrics.
Wanda++: Pruning Large Language Models via Regional Gradients (2025.findings-acl)

Copied to clipboard

Challenge: Existing pruning methods suffer from accuracy degradation without full-model sparsity-aware fine-tuning.
Approach: They propose a pruning framework that uses decoder-block-level regional gradients to improve pruning accuracy.
Outcome: The proposed pruning framework outperforms the state-of-the-art pruning frameworks by utilizing decoder-block-level regional gradients.
Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMs (2026.acl-long)

Copied to clipboard

Challenge: Existing approaches to personalize large language models (LLMs) rely on heuristic methods to compress user profiles but they ignore how LLMs process and prioritize different profile components.
Approach: They propose an attention-guided context compression framework that leverages attention feedback from a marking model to mark important personalization sentences and guides a compression model to generate task-relevant compressed user contexts.
Outcome: The proposed framework outperforms baselines across tasks, token limits, and settings while reducing token usage by 50 times.
When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models (2025.emnlp-main)

Copied to clipboard

Challenge: Large language models (LLMs) have achieved impressive performance across NLP tasks.
Approach: They propose to use long-context SFT to improve short-contemporary performance . they also decouple and analyze two key components, Multi-Head Attention and Feed-Forward Network .
Outcome: The proposed model improves short-context performance, contrary to pretraining.
CSS: A Large-scale Cross-schema Chinese Text-to-SQL Medical Dataset (2023.findings-acl)

Copied to clipboard

Challenge: a cross-domain text-to-SQL task aims to parse user questions into SQL on complete unseen databases . a single-domain task evaluates the performance on identical databases based on the same domain .
Approach: They propose a cross-domain text-to-SQL task that parses user questions into SQL on unseen databases.
Outcome: The proposed system can parse user questions into SQL on complete unseen databases.
CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space (2025.emnlp-main)

Copied to clipboard

Challenge: Embodied Question Answering (EQA) tasks are primarily focused on indoor environments, leaving the complexities of urban settings unexplored.
Approach: They propose a task where an embodied agent answers open-vocabulary questions in dynamic city spaces.
Outcome: The proposed agent achieves 60.7% of human-level answering accuracy compared to baselines . the proposed agent outperforms existing agents in open-ended city spaces .
Knowledge Stimulated Contrastive Prompting for Low-Resource Stance Detection (2022.findings-emnlp)

Copied to clipboard

Challenge: Stance Detection Tasks require background knowledge especially when there is no explicit target mentioned in text.
Approach: They propose a masked language prompt joint contrastive learning approach to stimulate the knowledge inherit from pre-trained models.
Outcome: The proposed model is effective in stance detection on three benchmarks.
QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models (2025.emnlp-main)

Copied to clipboard

Challenge: Large Language Models (LLMs) are quantized to lower precision to reduce memory cost and latency in inference.
Approach: They propose a quantized zeroth-order framework for fine-tuning Large Language Models (LLMs) using low-precision forward passes.
Outcome: The proposed method achieves comparable results to first-order methods in FP8 and superior accuracy in INT8 and INT4 training.
WALNUT: A Benchmark on Semi-weakly Supervised Learning for Natural Language Understanding (2022.naacl-main)

Copied to clipboard

Challenge: Existing studies on weak supervision for NLU focus on a specific task or simulate weak supervision signals from ground-truth labels.
Approach: They propose a benchmark to advocate and facilitate research on weak supervision for NLU . they use document-level and token-level prediction tasks as examples .
Outcome: The proposed benchmark advocates and facilitates research on weak supervision for NLU tasks.
SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoning (2026.findings-acl)

Copied to clipboard

Challenge: Recent large reasoning models have shown exceptional performance on various tasks, but they consume excessive tokens even for simple queries, leading to resource waste and prolonged user latency.
Approach: They propose a self-adaptive reasoning strategy that automatically allocates budgets according to problem complexity and introduces GRPO for reinforcement learning to reduce output length.
Outcome: The proposed model achieves an average response length compression of 61% on math reasoning tasks while maintaining accuracy.
CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations (2022.coling-1)

Copied to clipboard

Challenge: Pre-trained language models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding.
Approach: They propose a Chinese pre-trained language model that implicitly encodes words into characters . they propose 'contrastive learning over word' and 'character' representations to improve learning .
Outcome: The proposed model can encode words into fine-grained representations without modification of production pipelines.
Reinforcement Learning on Pre-Training Data (2026.acl-long)

Copied to clipboard

Challenge: Recent progress in large language models is driven by scaling of training compute through pre-training with nexttoken prediction (NTP) or post-training (RL) Pre-training using NTP enables models to acquire extensive knowledge and skills from general data, but it suffers from data inefficiency and catastrophic forgetting in continual learning settings.
Approach: They propose to scale training compute through pre-training with next-token prediction (NTP) or post-training by scaling reinforcement learning (RL) to improve learning from general data.
Outcome: Experiments on multiple benchmarks and models show that the proposed approach improves continual pre-training and provides a strong foundation for post-training on Qwen3-8B-Base.
X-LeBench: A Benchmark for Extremely Long Egocentric Video Understanding (2025.findings-emnlp)

Copied to clipboard

Challenge: Existing benchmark datasets focus on short to moderately long videos, leaving a substantial gap in evaluating extensive, ultra-long egocentric video recordings.
Approach: X-LeBench is a benchmark dataset designed to evaluate long egocentric video recordings . it uses a life-logging pipeline to produce realistic, coherent daily plans .
Outcome: X-LeBench is a new benchmark dataset designed to evaluate long-form egocentric video understanding . the approach produces realistic, coherent daily plans aligned with real-world video data .
Adversarial Knowledge Stimulated Contrastive Prompting for Few-shot Language Learners (2023.findings-acl)

Copied to clipboard

Challenge: Prompt-based fine-tuning has boosted performance of Pre-trained language models on few-shot Natural Language Understanding (NLU) tasks by employing task-specific prompts.
Approach: They propose a Cloze-driven prompt framework for prompt tuning that implicitly stimulates knowledge from pre-trained language models.
Outcome: The proposed framework outperforms state-of-the-art for prompt-based fine-tuning on few-shot NLU tasks.
Multimodal Dialogue Response Generation (2022.acl-long)

Copied to clipboard

Challenge: Existing studies focus on multimodal dialogue models but neglect generation methods.
Approach: They propose a multimodal dialogue response generation task which requires multimodal dialogs containing both texts and images which are difficult to obtain.
Outcome: Experiments show that the proposed model can generate informative text and high-resolution image responses.
BERT-QE: Contextualized Query Expansion for Document Re-ranking (2020.findings-emnlp)

Copied to clipboard

Challenge: Existing methods to expand query use pseudo relevance feedback (PRF) but they are under-equipped to evaluate the relevance of information pieces used for expansion.
Approach: They propose a query expansion model that leverages the BERT model to select relevant document chunks for expansion.
Outcome: The proposed model significantly outperforms existing models on the TREC Robust04 and GOV2 test collections.
MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark (2024.findings-acl)

Copied to clipboard

Challenge: Recent advances in large language models have showcased significant improvements in mathematics, but traditional benchmarks like GSM8k offer a unidimensional perspective.
Approach: MathBench is a benchmark that rigorously assesses the mathematical capabilities of large language models.
Outcome: MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills.
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2025.acl-long)

Copied to clipboard

Challenge: Recent advances in multimodal large language models have led to progress in tackling complex reasoning tasks that combine textual and visual information.
Approach: They introduce a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark.
Outcome: The proposed model performs lower on MMMU-Pro than on the previous benchmark, ranging from 16.8% to 26.9%.
Training-Free Test-Time Contrastive Learning for Large Language Models (2026.findings-acl)

Copied to clipboard

Challenge: Existing training-free alternatives to training-based models are static or depend on external guidance.
Approach: They propose a training-free adaptation framework that enables a frozen LLM to improve online by distilling supervision from its own inference experiences.
Outcome: The proposed framework outperforms existing test-time adaptation methods under online evaluation.
Everyone is unique: Towards Behaviorally Heterogeneous Negotiation Dialogue Systems for Debt Collection (2026.acl-long)

Copied to clipboard

Challenge: Existing models that assume users to be static, rational agents with fixed preferences fail to capture rich behavioral heterogeneity in real-world debt collection scenarios.
Approach: They propose a public persona-enriched debt collection benchmark that highlights behavioral heterogeneity in negotiation.
Outcome: The proposed benchmark outperforms existing models in realistic scenarios using 16 state-of-the-art LLMs.
MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples (2025.coling-main)

Copied to clipboard

Challenge: Existing preference optimization methods such as DPO and KTO are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data.
Approach: They propose an algorithm that leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data.
Outcome: The proposed algorithm outperforms DPO, ORPO, and SimPO on MT-Bench and Arena-Hard.
Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark (2026.acl-long)

Copied to clipboard

Challenge: Existing evaluations treat visual understanding and generation in isolation or overlook tasks that inherently couple them.
Approach: They propose a benchmark that examines the bidirectional synergy between generation and understanding across eight reasoning-centric domains.
Outcome: The proposed model systematically unfolds the bidirectional synergy between generation and understanding across eight reasoning-centric domains.
Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)

Copied to clipboard

Challenge: Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction.
Approach: They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance.
Outcome: The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies.
Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach (2025.acl-long)

Copied to clipboard

Challenge: Large Language Models (LLMs) enhanced with external contexts face challenges in handling imperfect evidence.
Approach: They propose a framework that can balance internal knowledge with external contexts . they propose gating mechanisms and low-rank representation adapters to adjust hidden representations based on a lightweight intervention function .
Outcome: The proposed model can effectively balance internal knowledge with external context, similar to human cognitive processes.
The Strength of the Weakest Supervision: Topic Classification Using Class Labels (N19-3)

Copied to clipboard

Challenge: a topic classifier can understand only class labels when training for tasks that require a large amount of labeled documents.
Approach: They propose an algorithm that can initialize a topic classifier using only class labels . they propose a method that combines word embedding and naive Bayes classification .
Outcome: The proposed approach saves significant initial labeling effort by providing a "warm start" the proposed approach can be fine-tuned with more labeled documents to reach a certain performance level.
Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models (2026.findings-acl)

Copied to clipboard

Challenge: Recent advances in Generative Reward Models have demonstrated that scaling the length of Chain-of-Thought reasoning enhances reliability of evaluation.
Approach: They propose a framework that reconfigures raw rationales into structured Breadth-CoT and Depth-Co T through a modular synthesis pipeline.
Outcome: The proposed framework surpasses open-source RMs by an average of 8.2%.
Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Models (2025.findings-emnlp)

Copied to clipboard

Challenge: Low-rank tensor compression techniques are used for over-parameterized neural networks, but their applications to compress pre-trained LLMs for downstream tasks remain challenging due to the high-rank nature of pre-training data.
Approach: They propose sparse augmented tensor networks to enhance low-rank tenorized LLMs . they also propose a framework that enables full model compression .
Outcome: The proposed framework improves accuracy and efficiency in tensorized language models.
MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models (2025.emnlp-main)

Copied to clipboard

Challenge: Large language models (LLMs) have demonstrated exceptional capabilities across diverse tasks, but their fine-tuning requires significant memory, posing challenges for resource-constrained environments.
Approach: They propose a ZO-based framework that eliminates the need for backpropagation and provides a memory-efficient alternative to backprograming.
Outcome: The proposed framework surpasses first-order methods in performance and accuracy.
Social-aware Sparse Attention Network for Session-based Social Recommendation (2022.findings-emnlp)

Copied to clipboard

Challenge: Existing methods for predicting the next item for an anonymous session do not capture user preferences and noisy irrelevant interactions.
Approach: They propose to use social networks and historical sessions to provide personalized recommendations for the current session.
Outcome: The proposed model outperforms existing models on two benchmark datasets.
Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks (2023.emnlp-main)

Copied to clipboard

Challenge: Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape . established automatic evaluation metrics are poor surrogates, correlating weakly with human judgement.
Approach: They propose to use both automatic and human evaluation to evaluate generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction.
Outcome: The proposed model outperforms many popular models according to human reviewers on the majority of metrics, while scoring much worse when using classic automatic evaluation metrics.
T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step (2024.acl-long)

Copied to clipboard

Challenge: Existing studies evaluate the tool utilization ability of large language models based on the final output or only consider the single-step tool calling.
Approach: They propose a new approach to evaluate the tool utilization capability of large language models (LLMs) they decompose the tool usage into multiple sub-processes, including instruction following, planning, reasoning, retrieval, understanding, and review.
Outcome: The proposed model exhibits consistency with the outcome-oriented evaluation and provides a more fine-grained analysis of the capabilities of LLMs.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations