Papers by Fang Sun

39 papers
Model Composition for Multimodal Large Language Models (2024.acl-long)

Copied to clipboard

Challenge: Existing methods for creating versatile MLLMs rely on joint training with paired instruction data, which is resource-intensive and challenging to extend to new modalities.
Approach: They propose a new paradigm for multimodal large language models by reusing modality encoders and merging LLM parameters.
Outcome: The proposed model retains the modal understanding capabilities of each original model.
Context-DPO: Aligning Language Models for Context-Faithfulness (2025.findings-acl)

Copied to clipboard

Challenge: Context-DPO is the first alignment method specifically designed to enhance contextfaithfulness for large language models.
Approach: They propose a benchmark that simulates Retrieval-Augmented Generation scenarios with knowledge conflicts to evaluate context-faithfulness.
Outcome: The proposed method improves LLMs' context-faithfulness by 35% to 280% over open-source models.
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021.naacl-main)

Copied to clipboard

Challenge: Existing pre-trained models suffer from slow inference speed due to cross-modal attention in transformer architecture.
Approach: They propose a multimodal approach that accelerates the inference time of ITR by thousands of times . they extract pre-cached feature indexes offline and employ instant dot-product matching online .
Outcome: The proposed approach outperforms existing models that consume 1000 times magnitude of computational hours using the same features.
Cluster-Former: Clustering-based Sparse Transformer for Question Answering (2021.findings-acl)

Copied to clipboard

Challenge: Existing models for encoding long sequences in deep learning suffer from high latency and memory demands.
Approach: They propose a clustering-based sparse Transformer framework to perform attention across chunked sequences.
Outcome: The proposed framework achieves state-of-the-art on several major QA benchmarks.
If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs (2026.acl-long)

Copied to clipboard

Challenge: Existing benchmarks for large language models (LLMs) fail to capture these dynamics, focusing on static, open-ended evaluations.
Approach: They propose a benchmark to assess lifelong learning in large language models . they use two episodic datasets rich in narrative structure and character interactions .
Outcome: Experiments on LLMs show that non-parametric methods outperform parametric ones in managing stateful learning.
D-Artemis: A Deliberative Cognitive Framework for Mobile GUI Multi-Agents (2026.findings-acl)

Copied to clipboard

Challenge: Graphical User Interface (GUI) agents aim to automate a wide spectrum of human tasks by emulating user interaction.
Approach: They propose a deliberative framework that leverages a fine-grained tip retrieval mechanism to inform its decision-making process.
Outcome: The proposed framework achieves SOTA among open-source general models on AndroidWorld and ScreenSpot-V2 . it leverages a fine-grained, app-specific tip retrieval mechanism to inform its decision-making process .
Contrastive Distillation on Intermediate Representations for Language Model Compression (2020.emnlp-main)

Copied to clipboard

Challenge: Existing methods to compress language models use a simple L_2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one.
Approach: They propose a method that uses knowledge distillation to distill knowledge through intermediate layers of the teacher via a contrastive objective.
Outcome: The proposed method outperforms state-of-the-art methods on the GLUE benchmark.
Listening Like Humans: Semantics-Guided Noise-Robust Multimodal Speech Recognition (2026.acl-long)

Copied to clipboard

Challenge: Severe acoustic degradation results in unreliable ASR outputs . et al., 2024b): critical concerns regarding reliability and fairness of ASR .
Approach: They propose a multimodal framework that reframes ASR as semantics-guided speech reconstruction.
Outcome: The proposed framework achieves an average reduction in WER while also attaining 98.71% BERTScore and 96.7% USE over advanced baselines.
Human-in-the-loop Robotic Grasping Using BERT Scene Representation (2022.coling-1)

Copied to clipboard

Challenge: Existing approaches for robotic grasping in cluttered scenes are expensive and lack structure information.
Approach: They propose a human-in-the-loop framework for robotic grasping in cluttered scenes . they substitute scene-graph representation with a text representation of the scene using BERT .
Outcome: The proposed framework outperforms object-agnostic and scene-graph based methods on robots and physical robots.
Backdooring Neural Code Search (2023.acl-long)

Copied to clipboard

Challenge: Neural code search models are used to find code snippets from online repositories . however, their security aspect is rarely studied .
Approach: They propose to use off-the-shelf code snippets from online repositories to find desired code . they propose to inject a backdoor into neural code search models which return buggy code if attacker modifies one variable/function name .
Outcome: The proposed attack outperforms baselines on two neural code search models by 60%.
Beyond Quantity: Trajectory Diversity Scaling for Code Agents (2026.findings-acl)

Copied to clipboard

Challenge: Code large language models (LLMs) are becoming tool-interactive agents . quantity-centric scaling exhibits an early bottleneck that underutilizes trajectory data . et al.: a new approach to scale trajectory diversity improves tool-use generalization .
Approach: They propose a Trajectory Diversity Scaling-based data synthesis framework for code agents that scales performance through diversity rather than raw volume.
Outcome: Experiments on general tool-use benchmarks and code agent tasks show that TDScaling improves tool-user generalization and inherent coding proficiency.
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding (2024.emnlp-main)

Copied to clipboard

Challenge: Speculative decoding is a widely used method that accelerates the generation process of large language models (LLMs) drafting efficiency has become a bottleneck in the final speedup of speculative drafting, therefore generating longer drafts at less cost can lead to better speedup.
Approach: They propose a method that uses existing model to drafting and target LLM to verify draft in a low-cost parallel manner.
Outcome: The proposed method can achieve speedups of up to 2.4 over speculative decoding and 3.9 over vanilla decoding without fine-tuning draft and target models.
SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration (2026.findings-acl)

Copied to clipboard

Challenge: Existing benchmarks emphasize general-domain retrieval or static scientific question answering . SciExplore focuses on scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis tasks.
Approach: They propose a benchmark to evaluate scientific information-seeking and reasoning capabilities of LLMs and agents.
Outcome: The new benchmark assesses the capabilities of state-of-the-art LLMs and agents in scientific research workflows.
T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generation (2026.findings-acl)

Copied to clipboard

Challenge: Text-to-image (T2I) generative models have demonstrated exceptional capability in synthesizing high-quality images from textual prompts.
Approach: They propose a benchmark to explore the knowledge-driven reasoning capabilities of T2I models.
Outcome: The proposed benchmark examines the knowledge-driven reasoning capabilities of T2I models.
FocusLLM: Precise Understanding of Long Context by Dynamic Condensing (2025.acl-long)

Copied to clipboard

Challenge: Existing context condensing methods cannot accurately understand the full context, as there is a considerable amount of information loss in the condensed process.
Approach: They propose a framework to extend the fixed context length of any decoder-only LLM by distilling crucial information from long sequences.
Outcome: The proposed framework extends the fixed context length of any decoder-only LLM, allowing it to focus on relevant information from very long sequences.
Cross-Thought for Sentence Encoder Pre-training (2020.emnlp-main)

Copied to clipboard

Challenge: Existing models to pretrain sentence encoders with large unlabeled corpus are lacking in linguistic information retrieval.
Approach: They propose a novel approach to pre-training sequence encoder using transformers . they propose to train a Transformer-based sequence encoded over a large set of short sequences based on a set of masked words .
Outcome: The proposed approach outperforms state-of-the-art encoders on hotpotQA by improving intermediate information retrieval performance.
NL2Lean: Translating Natural Language into Lean 4 through Multi-Aspect Reinforcement Learning (2025.emnlp-main)

Copied to clipboard

Challenge: Existing formal proof assistants rely on instruction tuning and lack fine-grained structural and semantic alignment.
Approach: They propose a reinforcement learning framework that enables LLMs to translate natural language into formal language such as Lean 4 . they use a model with basic translation ability to refine the model's reinforcement learning .
Outcome: The proposed method outperforms baseline models on NL-to-Lean 4 tasks.
Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching (2022.findings-naacl)

Copied to clipboard

Challenge: Existing studies have shown that cross-lingual knowledge distillation can improve the performance of pre-trained models for cross-linguistic similarity matching tasks.
Approach: They propose a multi-stage distillation framework for constructing a small-size but high-performance cross-lingual model using contrastive learning, bottleneck, and parameter recurrent strategies.
Outcome: The proposed model can compress the size of XLM-R and MiniLM by more than 50% while the performance is only reduced by about 1%.
Leveraging Knowledge in Multilingual Commonsense Reasoning (2022.findings-acl)

Copied to clipboard

Challenge: Commonsense reasoning is a language-agnostic process, but most comprehensive knowledge sources are limited to a small number of languages, especially English.
Approach: They propose to use English as a pivot language to integrate commonsense reasoning into models using a translate-retrieve-translate strategy.
Outcome: The proposed model outperforms the state-of-the-art on the XCSR benchmarks.
GUICourse: From General Vision Language Model to Versatile GUI Agent (2025.acl-long)

Copied to clipboard

Challenge: Graphical User Interfaces (GUIs) are a pivotal medium for human-computer interaction.
Approach: They propose a series of datasets for training visual-based GUI agents using general VLMs.
Outcome: The proposed GUICourse datasets show that even a small-sized GUI agent performs better on GUI tasks.
Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contexts (2025.emnlp-main)

Copied to clipboard

Challenge: Recent models have extended Corresponding Author. context lengths to millions of tokens while maintaining reasoning and comprehension capabilities.
Approach: They propose a benchmark to evaluate the ability of large language models to extract sequential information items from long contexts.
Outcome: The proposed model achieves maximum accuracy of 63.50% on six well-known LLMs.
Hierarchical Graph Network for Multi-hop Question Answering (2020.emnlp-main)

Copied to clipboard

Challenge: Existing multi-hop question answering models focus on multi-level reasoning across multiple documents or paragraphs.
Approach: They propose a hierarchical graph network that aggregates clues from scattered texts . they use a set of contextual encoders to initialize nodes on different levels of granularity .
Outcome: The proposed model outperforms existing multi-hop QA approaches on the HotpotQA benchmark.
AMPO: Automatic Multi-Branched Prompt Optimization (2024.emnlp-main)

Copied to clipboard

Challenge: Existing prompt engineering techniques are limited to producing single flow instructions, struggling with handling diverse patterns.
Approach: They propose an automatic prompt optimization method that iteratively develops a multi-branched prompt using failure cases as feedback.
Outcome: The proposed method achieves the best results across five tasks and demonstrates significant optimization efficiency due to adoption of a minimal search strategy.
AutoFigure-Edit: Generating Editable Scientific Illustrations via Reference-Guided Styling (2026.acl-demo)

Copied to clipboard

Challenge: Existing automated systems for scientific illustrations are limited in editability, stylistic controllability, and efficiency.
Approach: They propose an end-to-end system that generates fully editable scientific illustrations from long-form scientific text while enabling flexible style adaptation through user-provided reference images.
Outcome: The proposed system generates fully editable scientific illustrations from long-form scientific texts while enabling flexible style adaptation through user-provided reference images.
UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervision (2026.acl-long)

Copied to clipboard

Challenge: Unified Multimodal Models have achieved remarkable success in cross-modal comprehension, but a gap persists in their ability to translate internal knowledge into faithful and controllable synthesis.
Approach: They propose a self-improvement framework that partitions a single UMM into three collaborative roles: Proposer, Solver, and Judge.
Outcome: The proposed framework improves on TIIF, DPG, CompBench and UniCycle benchmarks.
Automated Molecular Concept Generation and Labeling with Large Language Models (2025.coling-main)

Copied to clipboard

Challenge: Concept-based models lack explainability and need predefined concepts and manual labeling in molecular science.
Approach: They propose a framework that leverages Large Language Models to generate and label predictive molecular concepts without human input.
Outcome: The proposed framework outperforms existing models on several benchmarks while maintaining explainability and allowing easy intervention.
MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation (2026.findings-acl)

Copied to clipboard

Challenge: Large language models suffer performance degradation when user instructions and context are distributed over multiple conversational turns.
Approach: They propose a framework that condenses chat history in the background without disrupting the user experience.
Outcome: The proposed framework reduces token counts by up to 72% in 10-turn dialogues while remaining robust to distractors and irrelevant turns.
Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation (2025.findings-acl)

Copied to clipboard

Challenge: Recent studies have evaluated and shown limitations in specific capabilities such as visual understanding, but a systematic evaluation of VLMs’ fundamental WM abilities remains absent.
Approach: They propose a framework that assesses perception and prediction to provide an atomic evaluation of VLMs as WMs.
Outcome: The proposed framework assesses perception and prediction abilities on 15 latest VLMs and compares them to human-level models.
Train in Vain: Functionality-Preserving Poisoning to Prevent Unauthorized Use of Code Datasets (2026.findings-acl)

Copied to clipboard

Challenge: Existing methods for dataset poisoning require full-dataset poison, which breaks code compilability.
Approach: They propose a functionality-preserving poisoning approach that injects short, compilable weak-use fragments into executed code paths.
Outcome: The proposed method contaminates 10% of the dataset while maintaining 100% compilability and functional correctness.
Why Are We Moral? An LLM-based Agent Simulation Approach to the Study of Moral Evolution (2026.acl-long)

Copied to clipboard

Challenge: Existing models of moral evolution must abstract away cognitive processes . et al. (2017): evolution of morality presents a puzzle: natural selection favors selfish .
Approach: They propose an LLM-based agent simulation framework that manipulates cognitive factors to understand moral evolution.
Outcome: The proposed model exploits cognitive realism to explore moral evolution in a hunter-gatherer society.
Infinite Babble: Inflating 3D Vision-Language Model Inference Overhead via Adversarial Geometric Perturbation (2026.findings-acl)

Copied to clipboard

Challenge: 3D Vision-Language Models (VLMs) are critical cognitive backbone for spatial intelligence, but their reliance on autoregressive decoding introduces a fundamental vulnerability regarding inference efficiency.
Approach: They propose a framework that triggers computational and economic exhaustion in 3D-VLMs by injecting imperceptible noise that forces the model into a state of pathological verbosity.
Outcome: The proposed framework amplifies output length and energy consumption by up to 6.45, demonstrating a potent capability to drain system resources.
QuantumQA: Enhancing Scientific Reasoning via Physics-Consistent Dataset and Verification-Aware Reinforcement Learning (2026.acl-long)

Copied to clipboard

Challenge: Large language models lack reliability in scientific domains that require strict adherence to physical constraints.
Approach: They propose a large-scale dataset constructed via a task-adaptive strategy and a hybrid verification protocol that combines deterministic solvers with semantic auditing to guarantee scientific rigor.
Outcome: The proposed model outperforms baselines and general-purpose preference models and is competitive with proprietary models.
GraphICL: Unlocking Graph Learning Potential in LLMs through Structured Prompt Design (2025.findings-naacl)

Copied to clipboard

Challenge: Text-Attributed Graphs (TAGs) are a powerful tool for understanding complex systems and relationships.
Approach: They propose a benchmark to evaluate large language models for graph-structured data using prompts.
Outcome: The proposed benchmark outperforms state-of-the-art graph LLMs and graph neural networks on graph learning tasks without training.
MQuinE: a Cure for “Z-paradox” in Knowledge Graph Embedding (2024.emnlp-main)

Copied to clipboard

Challenge: Existing knowledge graph embedding models suffer from Z-paradox, a deficiency in expressiveness . Embedding-based models map each entity and relation into a vector or matrix .
Approach: They propose a new knowledge graph embedding model that does not suffer from Z-paradox while preserves strong expressiveness to model various relation patterns with theoretical justification.
Outcome: The proposed model outperforms existing models on link prediction tasks while maintaining strong expressiveness.
LastingBench: Defend Benchmarks Against Knowledge Leakage (2025.findings-emnlp)

Copied to clipboard

Challenge: Existing methods to detect and safeguard LLMs against knowledge leakage fail to address the long-term challenge of mitigating it.
Approach: They propose a method to reinforce and safeguard existing benchmarks against knowledge leakage by perturbation-based detection and counterfactual rewriting to disrupt memorization while preserving original intent.
Outcome: The proposed method reduces memorization effects in long-context QA benchmarks, providing a more accurate assessment of model reasoning and generalization abilities.
SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesis (2025.findings-emnlp)

Copied to clipboard

Challenge: Existing approaches to deep search training lack high-quality training trajectories, prohibitive computational costs and lack of high-fidelity training data.
Approach: They propose a framework that synthesizes high-quality training data by simulating real user interactions in live web search environments.
Outcome: The proposed framework synthesizes high-quality training data by simulating user interactions in live web search environments.
JTPRO: A Joint Tool–Prompt Reflective Optimization Framework for Language Agents (2026.findings-acl)

Copied to clipboard

Challenge: Large language model agents struggle with ambiguous tool descriptions and underspecified tool schemas that ignore tool-specific nuances.
Approach: They propose a framework for improving tool-calling reliability in trace-supervised settings by rolling out-driven reflection.
Outcome: The proposed framework outperforms baselines and reflective prompt optimizers by 5%–20% on OSR.
AiraXiv: An AI-Driven Open-Access Platform for Human and AI Scientists (2026.acl-demo)

Copied to clipboard

Challenge: Recent advances in artificial intelligence (AI) have accelerated the growth of both human-authored and AI-generated research outputs.
Approach: They propose an AI-driven open-access platform built on open preprints, AI-augmented analysis and review, and reader feedback.
Outcome: The proposed platform supports human scientists through an interactive UI and AI scientists through Model Context Protocol (MCP)-based interactions.
Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data (2022.acl-long)

Copied to clipboard

Challenge: Experimental results show that REtrieving from the traINing datA only can lead to significant gains on multiple NLG and NLU tasks.
Approach: They propose to retrieve training instances from traINing datA and concatenate them with input to generate output.
Outcome: The proposed method achieves state-of-the-art results on XSum, BigPatent, and CommonsenseQA.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations