Papers by Liangming Pan

43 papers
Exploring Question-Specific Rewards for Generating Deep Questions (2020.coling-main)

Copied to clipboard

Challenge: Recent question generation approaches use the sequence-to-sequence framework to optimize the log likelihood of ground-truth questions using teacher forcing.
Approach: They propose to optimize for QG-specific objectives via reinforcement learning to improve question quality.
Outcome: The proposed model improves the fluency, relevance, and answerability of generated questions.
Gödel Agent: A Self-Referential Agent Framework for Recursively Self-Improvement (2025.acl-long)

Copied to clipboard

Challenge: Existing agentic systems cannot search the whole design space due to the restriction of human-designed components.
Approach: They propose a Gödel Agent framework that allows agents to recursively improve themselves without relying on fixed algorithms or fixed algorithms.
Outcome: The proposed framework surpasses manual crafted agents in performance, efficiency, and generalizability.
Fact-Checking Complex Claims with Program-Guided Reasoning (2023.acl-long)

Copied to clipboard

Challenge: Fact-checking real-world claims often requires collecting multiple pieces of evidence and complex multi-step reasoning.
Approach: They propose a novel fact-checking model that decomposes complex claims into simpler sub-tasks that can be solved using a shared library of specialized functions.
Outcome: The proposed model outperforms seven baselines on two fact-checking datasets and has explicit output programs that benefit human debugging.
Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusion (2024.findings-acl)

Copied to clipboard

Challenge: Existing dynamic topic models lack the ability to reveal the evolution of topics . Existing models suffer from repetitive topic and unassociated topic issues .
Approach: They propose a new evolution-tracking contrastive learning method that builds the similarity relations among dynamic topics and an unassociated word exclusion method to avoid unassociated topics.
Outcome: The proposed model outperforms state-of-the-art models on downstream tasks and is robust to evolution intensities.
Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failures (2026.acl-long)

Copied to clipboard

Challenge: Recent research has shown that reinforcement learning can elicit intriguing emergent reasoning behaviors.
Approach: They propose a comprehensive survey of the mechanistic understanding of large reasoning models . they organize findings into three core dimensions: 1) training dynamics, 2) reasoning mechanisms, and 3) unintended behaviors.
Outcome: This paper synthesizes the mechanistic understanding of large reasoning models into three dimensions . authors outline a roadmap for future studies including improved interpretability and methodologies .
KHANQ: A Dataset for Generating Deep Questions in Education (2022.coling-1)

Copied to clipboard

Challenge: Existing QG datasets are not suitable for educational question generation because the questions are not real questions asked by humans during learning.
Approach: They propose a dataset for question generation that contains 1,034 high-quality learner-generated questions seeking an in-depth understanding of the taught online courses in Khan Academy.
Outcome: The proposed dataset contains 1,034 high-quality learner-generated questions seeking an in-depth understanding of the taught online courses in Khan Academy.
QACheck: A Demonstration System for Question-Guided Multi-Hop Fact-Checking (2023.emnlp-demo)

Copied to clipboard

Challenge: Existing fact-checking systems lack transparency in their decision-making process, making it difficult for users to comprehend their reasoning process.
Approach: They propose a Question-guided Multi-hop Fact-Checking system which asks a series of questions critical for verifying a claim.
Outcome: The proposed model provides a comprehensive report detailing its reasoning process, guided by a sequence of questions, answer pairs, and the source of evidence supporting each question.
Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models (2024.findings-acl)

Copied to clipboard

Challenge: Known-unknown questions are characterized by high uncertainty due to the absence of definitive answers.
Approach: They introduce a dataset with known-unknown questions and establish a categorization framework to clarify the origins of uncertainty in such queries.
Outcome: The proposed model improved in distinguishing between known and unknown queries within open-ended question-answering scenarios.
CausalEval: Towards Better Causal Reasoning in Language Models (2025.naacl-long)

Copied to clipboard

Challenge: Large language models (LLMs) have been used for a variety of tasks, including problem-solving, decision-making, and understanding of the world.
Approach: They propose a review of existing methods aimed at enhancing LMs for causal reasoning . they categorize existing methods as reasoning engines or as helpers providing knowledge or data to traditional methods .
Outcome: The proposed methods perform better than existing methods on a range of tasks.
Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve Framework (2025.acl-long)

Copied to clipboard

Challenge: Existing systems fail to fully leverage the structure of logical tasks throughout the reasoning process, causing bottlenecks in efficiency and efficacy.
Approach: They propose a logic-complete reasoning framework, Aristotle, which integrates symbolic expressions and logical rules into the entire reasoning process.
Outcome: The proposed framework outperforms state-of-the-art reasoning frameworks in accuracy and efficiency.
Expertise Style Transfer: A New Task Towards Better Communication between Experts and Laymen (2020.acl-main)

Copied to clipboard

Challenge: a curse of knowledge can impede communication between experts and laymen.
Approach: They propose a task of expertise style transfer to automate the communication between experts and laymen using an annotated dataset.
Outcome: The proposed task simplifies professional language and improves accuracy and expertise level of laymen descriptions using simple words.
Zero-shot Fact Verification by Claim Generation (2021.acl-short)

Copied to clipboard

Challenge: Existing methods for fact verification require large datasets, which can be expensive.
Approach: They propose a framework for training a robust fact verification model by using automatically generated claims that can be supported, refuted, or unverifiable from evidence from Wikipedia.
Outcome: The proposed framework reduces the demand for human-annotated training data and improves a model's F1 from 50% to 77%, equivalent in performance to 2K+ manually-curated examples.
AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge (2025.acl-long)

Copied to clipboard

Challenge: Existing studies solve this challenge by updating benchmarks with newly collected data, but they fail to guarantee contamination-free evaluation as the newly collected knowledge may contain pre-existing knowledge.
Approach: They propose an automated anti-leakage benchmarking framework that builds and updates benchmarks without human labor instead of using newly collected data.
Outcome: The proposed framework significantly reduces the cost of benchmark maintenance to accommodate emerging LLMs.
InductionBench: LLMs Fail in the Simplest Complexity Class (2025.acl-long)

Copied to clipboard

Challenge: Existing benchmarks focus on deductive reasoning, largely overlooking inductive reasoning.
Approach: They propose a benchmark to evaluate the inductive reasoning ability of large language models.
Outcome: The proposed benchmark demonstrates that even the most advanced modelw struggle to master the simplest complexity classes within the subregular hierarchy of functions.
On the Risk of Misinformation Pollution with Large Language Models (2023.findings-emnlp)

Copied to clipboard

Challenge: a recent study demonstrates that large language models can be misused for generating credible-sounding misinformation . however, the ability to produce credible text raises concerns regarding their potential misuse .
Approach: They propose three defense strategies to mitigate misinformation generated by Large Language Models . they propose a threat model and simulate potential misuse scenarios .
Outcome: The proposed defense strategies have shown promising results, albeit with costs.
SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented Generation (2025.acl-long)

Copied to clipboard

Challenge: Adaptive Retrieval-Augmented Generation (RAG) is an effective strategy to alleviate hallucination of large language models (LLMs).
Approach: They propose a novel adaptive RAG model that extracts self-aware uncertainty of large language models from their internal states and invokes retrieval accordingly.
Outcome: The proposed model outperforms existing adaptive RAG methods on complex and simple Question Answering datasets.
Unsupervised Multi-hop Question Answering by Question Generation (2021.naacl-main)

Copied to clipboard

Challenge: Existing training data for multi-hop question answering (QA) is time-consuming and resource-intensive.
Approach: They propose an unsupervised framework that generates human-like multi-hop training data from homogeneous and heterogeneously data sources.
Outcome: The proposed framework achieves 61% and 83% of the supervised learning performance for the HybridQA and HotpotQA datasets.
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers (2024.findings-emnlp)

Copied to clipboard

Challenge: Large language models generate naturally sounding answers over a broad range of human inquiries, but they often generate answers that contradict real-world facts.
Approach: They propose a framework for annotating and evaluating the factuality of large language models . they propose 'factcheck-bench' which provides a multi-stage annotation scheme .
Outcome: The proposed framework outperforms several popular LLM fact-checkers in claim, sentence, and document levels.
INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback (2023.emnlp-main)

Copied to clipboard

Challenge: Existing methods to evaluate the quality of language generation do not provide explicit explanation of their verdicts.
Approach: They propose a fine-grained explainable evaluation metric for text generation that harnesses human instruction and implicit knowledge of GPT-4 to fine-tune it.
Outcome: The proposed model outperforms all other unsupervised metrics on translation, captioning, data-to-text, and commonsense generation tasks.
MultiAgent Collaboration Attack: Investigating Adversarial Attacks in Large Language Model Collaborations via Debate (2024.findings-emnlp)

Copied to clipboard

Challenge: Large Language Models (LLMs) have shown exceptional results when working individually, and have reduced parameter size and inference times.
Approach: They evaluate the behavior of a network of models collaborating through debate under the influence of an adversary and examine inference-time methods to generate more compelling arguments.
Outcome: The proposed model-based model-driven analysis shows that the model-led model-mediated debates generate more compelling arguments and provide a defensive strategy.
Exploring and Evaluating Attributes, Values, and Structures for Entity Alignment (2020.emnlp-main)

Copied to clipboard

Challenge: Entity alignment (EA) aims at building a Knowledge Graph (KG) of rich content by linking the equivalent entities from various KGs.
Approach: They propose to use an attributed value encoder to partition a Knowledge Graph into subgraphs to model the various types of attribute triples efficiently.
Outcome: The proposed method achieves significant improvements over 12 baselines in cross-lingual and monolingual datasets.
Investigating the Transferability of Code Repair for Low-Resource Programming Languages (2025.findings-naacl)

Copied to clipboard

Challenge: Large language models (LLMs) have shown remarkable performance on code generation tasks.
Approach: They investigate the benefits of distilling the ability to repair code for both high and low resource languages to determine if the techniques are also applicable in low resource settings.
Outcome: The proposed techniques are effective in high- and low-resource languages, but weak in low-level languages.
Interpreting the Robustness of Neural NLP Models to Textual Perturbations (2022.findings-acl)

Copied to clipboard

Challenge: Modern Natural Language Processing models are sensitive to input perturbations and their performance can decrease when applied to noisy data.
Approach: They propose to explain the extent to which a model is affected by an unseen textual perturbation by the learnability of the perturbation.
Outcome: The proposed model is better at identifying a perturbation (higher learnability) but worse at ignoring it (lower robustness).
TART: An Open-Source Tool-Augmented Framework for Explainable Table-based Reasoning (2025.findings-naacl)

Copied to clipboard

Challenge: Current Large Language Models lack ability to understand table structures and apply precise numerical reasoning.
Approach: They propose a tool-augmented reasoning framework for table-based tasks that integrates LLMs with specialized tools.
Outcome: The proposed framework improves on the TOOLTAB dataset, a benchmark for LLMs in table–tool integration.
Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution (2024.findings-acl)

Copied to clipboard

Challenge: Existing evaluation metrics and benchmarks to attribute large language models to structured knowledge are lacking.
Approach: They propose a task of Knowledge-aware Language Model Attribution that improves upon three core concerns with conventional attributed LMs.
Outcome: The proposed model improves upon core concerns with conventional attributed LMs.
A Survey on Detection of LLMs-Generated Content (2024.findings-emnlp)

Copied to clipboard

Challenge: Recent advances in large language models have led to an increase in synthetic content generation . the ability to detect LLMs-generated content has become of paramount importance .
Approach: They propose to provide a detailed overview of existing detection strategies and benchmarks, scrutinizing their differences and advocating for more adaptable and robust models to enhance detection accuracy.
Outcome: The proposed model will be able to detect human-written content in real time.
Modeling What-to-ask and How-to-ask for Answer-unaware Conversational Question Generation (2023.acl-long)

Copied to clipboard

Challenge: Existing methods to generate conversational question are naive and do not account for the answer span.
Approach: They propose a framework for generating a conversational question from a context.
Outcome: The proposed framework achieves state-of-the-art in two different settings compared to existing models . it uses a sentence as the rationale and extracts the answer span from it .
Doolittle: Benchmarks and Corpora for Academic Writing Formalization (2023.emnlp-main)

Copied to clipboard

Challenge: Existing methods of language refinement focus on narrow, specific linguistic features within isolated sentences, such as grammatical errors and improper word use.
Approach: They propose a task to improve the overall quality of academic writing at paragraph level by integrating automatic feedback into the training process.
Outcome: The proposed task improves the overall quality of formal academic writing at the paragraph level.
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023.findings-emnlp)

Copied to clipboard

Challenge: Large Language Models (LLMs) have shown human-like reasoning abilities but struggle with complex logical problems.
Approach: They propose a framework which integrates large language models with symbolic solvers to improve logical problem-solving by combining them with a self-refinement module.
Outcome: The proposed framework improves logical problem-solving by integrating LLMs with symbolic solvers.
CoHS-CQG: Context and History Selection for Conversational Question Generation (2022.coling-1)

Copied to clipboard

Challenge: Existing studies focus on single-turn question generation, but few studies have studied the challenges of multiturn QG.
Approach: They propose a two-stage conversational question generation framework that shortens the context and history of the input and calculates relevance scores.
Outcome: The proposed framework achieves state-of-the-art on CoQA in answer-aware and answer-unaware settings.
Semantic Graphs for Generating Deep Questions (2020.acl-main)

Copied to clipboard

Challenge: Existing research has focused on generating factoid questions relevant to one fact obtainable from a single sentence.
Approach: They propose a framework that first constructs a semantic-level graph and then encodes it by introducing an attention-based GGNN.
Outcome: The proposed framework captures the global structure of the document and facilitates reasoning over multiple facts.
The Knowledge Alignment Problem: Bridging Human and External Knowledge for Large Language Models (2024.findings-acl)

Copied to clipboard

Challenge: Large language models often ignore external knowledge to generate accurate answers . despite correct groundings, they can rely on wrong grounding or biases to hallucinate .
Approach: They propose a framework that integrates human and human user clarifications to improve knowledge alignment.
Outcome: The proposed framework improves model performance and mitigates hallucination by producing user-centered clarifications.
RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios (2025.acl-long)

Copied to clipboard

Challenge: RuleArena assesses the ability of large language models (LLMs) to follow complex, real-world rules in reasoning.
Approach: They propose a benchmark to evaluate the ability of large language models (LLMs) to follow complex, real-world rules in reasoning.
Outcome: The proposed benchmark covers airline baggage fees, NBA transactions, and tax regulations.
How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark (2025.emnlp-main)

Copied to clipboard

Challenge: Prior work has not explored the mechanisms underlying this sensitivity.
Approach: They propose a synthetic benchmark to evaluate Large Language Models’ reasoning robustness against systematically controlled irrelevant context (IC).
Outcome: The proposed model improves in-distribution and out-of-disttribution scenarios while training with strong distractors.
MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models (2023.emnlp-main)

Copied to clipboard

Challenge: Existing approaches to enhance Language Models fail to address diverse error types . generic feedback is a bottleneck for addressing diverse errors in reasoning chains .
Approach: They propose an iterative refinement framework that integrates multiple feedback modules . they propose to address errors in reasoning chains by integrating frozen LMs with external tools .
Outcome: The proposed framework improves performance in Mathematical Reasoning and Logical Entailment by 20% and 18% respectively.
SciAgent: Tool-augmented Language Models for Scientific Reasoning (2024.emnlp-main)

Copied to clipboard

Challenge: SciAgent surpasses other LLMs with the comparable size by more than 8.0% in absolute accuracy.
Approach: They propose a tool-augmented scientific reasoning setting that supplements LLMs with scalable toolsets and builds a benchmark to evaluate LLM’s abilities with tool assistance.
Outcome: The proposed setting augments LLMs with scalable toolsets and shifts the focus from pursuing an omniscient problem solver to a proficient tool-user.
SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables (2023.emnlp-main)

Copied to clipboard

Challenge: Current scientific fact-checking benchmarks exhibit several shortcomings, such as biases arising from crowd-sourced claims and an over-reliance on text-based evidence.
Approach: They present a dataset of 1.2K expert-verified scientific claims that require compositional reasoning for verification.
Outcome: The proposed model outperforms existing models in table-based pretraining models and large language models.
AKEW: Assessing Knowledge Editing in the Wild (2024.emnlp-main)

Copied to clipboard

Challenge: Recent Large Language Models (LLMs) have revolutionized the NLP field but their knowledge could become incorrect or outdated over time.
Approach: They propose a new practical benchmark for knowledge editing that covers three editing settings of knowledge updates: structured facts, unstructured texts as facts, and extracted triplets.
Outcome: The proposed method covers structured facts, unstructured texts as facts, and extracted triplets.
Towards Intrinsic Interpretability of Large Language Models: A Survey of Design Principles and Architectures (2026.acl-long)

Copied to clipboard

Challenge: Existing studies on explainable AI focus on post-hoc explanation methods that interpret trained models through external approximations.
Approach: They propose to categorize existing approaches into five design paradigms: functional transparency, concept alignment, representational decomposability, explicit modularization, and latent sparsity induction.
Outcome: The proposed approaches are categorized into five design paradigms: functional transparency, concept alignment, representational decomposability, explicit modularization, and latent sparsity induction.
LEDOM: Reverse Language Model (2026.acl-long)

Copied to clipboard

Challenge: Autoregressive language models are trained exclusively left-to-right, yet they are limited in their ability to factorize text.
Approach: They propose a purely reverse autoregressive language model that factorizes text as a product of left-to-right conditionals.
Outcome: The proposed model can be used to score forward outputs using reverse posterior estimates.
KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base (2022.acl-long)

Copied to clipboard

Challenge: Existing benchmarks for Complex KBQA lack compositional reasoning capabilities . Existing methods for Complex questions are poor in diversity or scale .
Approach: They propose a compositional programming language to represent the reasoning process of complex questions.
Outcome: The proposed dataset includes around 120K diverse natural language questions . it provides a compositional and interpretable programming language to represent the reasoning process of complex questions based on the proposed model .
Faithful Logical Reasoning via Symbolic Chain-of-Thought (2024.acl-long)

Copied to clipboard

Challenge: SymbCoT is a framework that integrates symbolic expressions and logic rules with CoT prompting.
Approach: They propose a Symbolic Chain-of-Thought framework that integrates symbolic expressions and logic rules with CoT prompting.
Outcome: The proposed framework improves on 5 standard datasets with symbolic expressions and rules . it shows that it is more faithful, flexible, and explainable than the current method .
ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models (2025.findings-emnlp)

Copied to clipboard

Challenge: Large language models excel at complex tasks by breaking down problems into structured reasoning steps.
Approach: They propose a hyperparameter-free conciseness score to guide models toward generating correct reasoning traces.
Outcome: The proposed method reduces token usage by 31x on simple problems while improving accuracy by 7% on the hardest problems.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations