Papers by Kaixin Ma
Tree-of-Evolution: Tree-Structured Instruction Evolution for Code Generation in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Data synthesis is a key research area in large language models (LLMs). |
| Approach: | They propose a framework that models code instruction synthesis process with a tree structure and optimization-driven evolution to alleviate constraints of unidirectional synthesis and randomness-driven generation. |
| Outcome: | The proposed framework outperforms open-weight code LLMs on five widely-used benchmarks. |
BRAINTEASER: Lateral Thinking Puzzles for Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | lateral thinking tasks require implicit and complex reasoning, relying on human-like commonsense mechanisms. |
| Approach: | They propose a lateral thinking benchmark to test models' ability to exhibit lateral reasoning and defy default commonsense associations. |
| Outcome: | The proposed model exhibits lateral thinking and defies default commonsense associations. |
Challenging Reading Comprehension on Daily Conversation: Passage Completion on Multiparty Dialog (N18-1)
Copied to clipboard
| Challenge: | Existing approaches to reading comprehension on multiparty dialogs have focused on children's stories or newswire. |
| Approach: | They propose a new corpus and a robust deep learning architecture for a task in reading comprehension on multiparty dialog. |
| Outcome: | The proposed model outperforms the state-of-the-art model on a different genre using bidirectional LSTM, showing a 13.0+% improvement for longer dialogs. |
WebRollback: Enhancing Web Agents with Explicit Rollback Mechanisms (2026.eacl-short)
Copied to clipboard
| Challenge: | Recent studies have adopted a greedy one-way search strategy to deal with dynamic web environments. |
| Approach: | They propose to integrate a rollback mechanism into web agents to allow them to revert back to a previous state in navigation trajectory. |
| Outcome: | The proposed method is able to revert back to a previous state in its navigation trajectory, allowing the models to directly control the search process. |
Bend but Don’t Break? Multi-Challenge Stress Test for QA Models (D19-58)
Copied to clipboard
| Challenge: | a gap remains in reasoning ability compared to a human, and performance tends to degrade when models are exposed to less-constrained tasks. |
| Approach: | They conduct extensive qualitative and quantitative analyses on the results of four models across four datasets . they relate common errors to model capabilities and discuss a way forward . |
| Outcome: | The proposed model performance is based on the results of four models across four datasets. |
AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large Language Models on Harmfulness (2025.acl-long)
Copied to clipboard
| Challenge: | Existing models that assess mLLMs on harmful meme understanding are inaccurate and lack accuracy. |
| Approach: | They propose a framework that adaptively probes the reasoning capabilities of mLLMs . their framework systematically reveals the varying performance of different target mllms a . |
| Outcome: | The proposed framework systematically reveals the performance of different target mLLMs. |
Coalescing Global and Local Information for Procedural Text Understanding (2022.coling-1)
Copied to clipboard
| Challenge: | Existing models for procedural text understanding have low precision or low recall . et al., 2012, pp. 106-106. |
| Approach: | They propose a model that builds entity- and timestep-aware input representations . they extend the model with additional output layers and integrate it into a story reasoning framework . |
| Outcome: | The proposed model achieves state-of-the-art on a popular procedural text understanding dataset and on 'story reasoning benchmark' it integrates the model with additional output layers and improves on the previous models. |
OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization (2025.acl-long)
Copied to clipboard
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Hongming Zhang, Tianqing Fang, Zhenzhong Lan, Dong Yu
| Challenge: | Existing studies focus on building text-only agents in synthetic environments where the reward signals are clearly defined. |
| Approach: | They propose a multimodal web agent that can autonomously conduct real-world exploration and improve itself after each iteration. |
| Outcome: | The proposed agent improves itself after each iteration, demonstrating strong performance across multiple test sets. |
Retrieval-augmented GUI Agents with Generative Guidelines (2025.emnlp-main)
Copied to clipboard
| Challenge: | GUI agents powered by vision-language models struggle with real-world tasks due to their complex nature and limited training data. |
| Approach: | They propose a lightweight vision-language model that leverages web tutorials at inferencetime to synthesize GUI agents. |
| Outcome: | The proposed agent outperforms baseline GUI agents and surpasses other inference baselines by 2.6% to 13.3% across two model sizes. |
Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposes lightweight updates to improve commonsense reasoning models . fine-tuning can cause models to overfit to task-specific data and forget knowledge gained during training . |
| Approach: | They propose to use lightweight models to update pre-trained language models to learn commonsense background knowledge. |
| Outcome: | The proposed models learn from commonsense reasoning datasets, but they are overfitted and limited generalized. |
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing web agents only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. |
| Approach: | They propose a large multimodal model-powered web agent that can complete user instructions end-to-end by interacting with real-world websites. |
| Outcome: | The proposed agent achieves 59.1% task success rate, surpassing both GPT-4 and WebVoyager setups. |
MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation approaches focus on mLLMs’ detection accuracy for binary classification tasks, which often fail to reflect the in-depth interpretive nuance of harmfulness across diverse contexts. |
| Approach: | They propose an agent-based arena-style evaluation framework that provides context-aware and unbiased assessment for mLLMs’ understanding of multimodal harmfulness. |
| Outcome: | The proposed framework reduces evaluation biases of judge agents and provides unbiased comparisons of mLLMs’ abilities to interpret multimodal harmfulness. |
Chain-of-Skills: A Configurable Model for Open-Domain Question Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Using customized retrieval models, model transferability and scalability are limited. |
| Approach: | They propose a modular retrieval model where individual modules correspond to key skills that can be reused across datasets. |
| Outcome: | The proposed model outperforms self-supervised retrievers in zero-shot evaluations and achieves state-of-the-art fine-tuned retrieval performance on NQ, HotpotQA and OTT-QA. |
MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique (2025.findings-emnlp)
Copied to clipboard
| Challenge: | e MM-CRITIC is a holistic benchmark for evaluating the critique ability of Large Multimodal Models (LMMs) covering 8 main task types and over 500 tasks, covering 4471 samples. |
| Approach: | They introduce a holistic benchmark for evaluating the critique ability of Large Multimodal Models across multiple dimensions: basic, correction, and comparison. |
| Outcome: | The proposed benchmark covers 8 main task types and over 500 tasks and is composed of 4471 samples. |
DivScene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenes (2025.findings-emnlp)
Copied to clipboard
Zhaowei Wang, Hongming Zhang, Tianqing Fang, Ye Tian, Yue Yang, Kaixin Ma, Xiaoman Pan, Yangqiu Song, Dong Yu
| Challenge: | Large Vision-Language Models (LVLMs) have achieved significant progress in tasks like visual question answering and document understanding. |
| Approach: | They introduce DivScene, a large-scale dataset with 4,614 houses across 81 scene types and 5,707 kinds of target objects. |
| Outcome: | The proposed dataset provides a much greater diversity of target objects and scene types than existing datasets, enabling a comprehensive task evaluation. |
Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing open-domain question answering methods rely on the retriever to gather all evidence in isolation, but our approach uses an intermediary module to perform a chain of reasoning over the retrieved set. |
| Approach: | They propose a new open-domain question answering framework that integrates an intermediary module into the current retriever-reader pipeline and integrates it into the model. |
| Outcome: | The proposed framework outperforms the state-of-the-art on two OTT-QA datasets with an exact match score of 47.3 (45% relative gain). |
WebEvolver: Enhancing Web Agent Self-Improvement with Co-evolving World Model (2025.emnlp-main)
Copied to clipboard
| Challenge: | Agent self-improvement, where agents train their underlying Large Language Model (LLM) on self-sampled trajectories, shows promising results but often stagnates in web environments due to limited exploration and under-utilization of pretrained web knowledge. |
| Approach: | They propose a co-evolving Large Language Model (LLM) that predicts the next observation based on current observation and action within the web environment. |
| Outcome: | The proposed framework shows that agents can perform better in real-world web environments without using any distillation from more powerful close-sourced models. |
Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering (D19-60)
Copied to clipboard
| Challenge: | Recent approaches on non-extractive commonsense QA show increased performance . attention-based injection seems to be preferable for knowledge integration . |
| Approach: | They propose to use attention-based injection to integrate knowledge into commonsense QA models. |
| Outcome: | The proposed methods show that attention-based injection is preferable for knowledge integration, and that the degree of domain overlap plays a crucial role in determining model success. |
From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms (2026.findings-acl)
Copied to clipboard
Jinghao Luo, Yuchen Tian, Chuxue Cao, Ziyang Luo, Hongzhan Lin, Kaixin Li, Chuyi Kong, Ruichao Yang, Jing Ma
| Challenge: | Large Language Models (LLMs)-based agents have fundamentally reshaped artificial intelligence . however, the inherent statelessness of LLMs hinders their ability to maintain logical consistency across complex, multi-step tasks . |
| Approach: | They propose a framework for LLM agent memory mechanisms that formalizes the development process into three stages: storage, reflection, and experience. |
| Outcome: | The proposed framework breaks the development process into three stages . it analyzes the need for long-range consistency, challenges in dynamic environments, and the ultimate goal of continual learning. |
Dense X Retrieval: What Retrieval Granularity Should We Use? (2024.emnlp-main)
Copied to clipboard
| Challenge: | a learned dense retrieval model is often overlooked when using a corpus for inference, resulting in a design choice of retrieval unit . granularity of retrievals is important for both retrieval and downstream tasks . |
| Approach: | They propose a retrieval unit for dense retrieval that uses propositions to index corpus . propositions are defined as atomic expressions within text, each encapsulating a distinct factoid . |
| Outcome: | The proposed retrieval unit outperforms passage-level units on retrieval and downstream tasks. |
Open Domain Question Answering with A Unified Knowledge Interface (2022.acl-long)
Copied to clipboard
| Challenge: | a retriever-reader framework is popular for open domain question answering . however, accessing heterogeneous knowledge sources through a unified interface remains unknown . |
| Approach: | They propose a retriever-reader framework that uses explicit knowledge to access heterogeneous knowledge sources through a unified interface. |
| Outcome: | The proposed framework can benefit from the expanded knowledge index, the authors show . their approach sets the single-model state-of-the-art on Natural Questions . |
MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Programming often involves translating detailed and complex specifications into code . current state-of-the-art models struggle to solve these problems, a new study shows . |
| Approach: | They propose a multi-modal coding dataset to evaluate algorithmic problem-solving skills in visually rich contexts. |
| Outcome: | The proposed model lacks powerful vision-code models due to the extreme demand for reasoning abilities. |
Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Standard RALMs often neglect their intrinsic knowledge due to the interference from retrieved information. |
| Approach: | They propose a new approach to improve robustness of RALMs by generating sequential reading notes for each retrieved document. |
| Outcome: | The proposed approach outperforms standard RALMs on four open-domain QA benchmarks. |