Papers by Hang Su
MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization (2025.findings-emnlp)
Copied to clipboard
Yinhong Liu, Jianfeng He, Hang Su, Ruixue Lian, Yi Nian, Jake W. Vincent, Srikanth Vishnubhotla, Robinson Piramuthu, Saab Mansour
| Challenge: | Multimodal Dialogue Summarization (MDS) is a critical task with wide-ranging applications. |
| Approach: | They propose a meta-evaluation benchmark for multimodal dialogue summarization based on image-sharing dialogues, corresponding summaries and human judgments . |
| Outcome: | The proposed framework is the first to identify and formalize key evaluation dimensions specific to MDS. |
APOLLO: An Optimized Training Approach for Long-form Numerical Reasoning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to generate reasoning programs that ignore the differences between facts treated all facts equally, leading to wrong punishment of programs that differed from the ground truth. |
| Approach: | They propose an optimized training framework for long-form numerical reasoning that incorporates a number-aware negative sampling strategy and consistency-based reinforcement learning to increase execution accuracy. |
| Outcome: | The proposed method improves the performance of long-form numerical reasoning on the FinQA and ConvFinQA leaderboards. |
FineSurE: Fine-grained Summarization Evaluation using LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text summarization evaluation do not correlate well with human judgments . evaluators that use Likert scale scores are limited in their ability to perform deeper analysis. |
| Approach: | They propose a fine-grained evaluator specifically tailored for the summarization task using large language models. |
| Outcome: | The proposed method improves on open-source and proprietary LLMs and shows better completeness and conciseness than existing methods. |
MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets (2024.naacl-long)
Copied to clipboard
Hossein Aboutalebi, Hwanjun Song, Yusheng Xie, Arshit Gupta, Lijia Sun, Hang Su, Igor Shalyminov, Nikolaos Pappas, Siffi Singh, Saab Mansour
| Challenge: | Existing approaches to augment textual dialogues with retrieved images pose privacy, diversity, and quality constraints. |
| Approach: | They propose a framework to augment text-only dialogues with diverse and high-quality images by using a diffusion model and a feedback loop. |
| Outcome: | The proposed framework is comparable to or better than baselines, with significant improvements in human evaluation, especially against retrieval baselines where the image database is small. |
Beta Distribution Guided Aspect-aware Graph for Aspect Category Sentiment Analysis with Affective Knowledge (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for aspect category sentiment analysis do not necessarily occur in a sentence. |
| Approach: | They propose a Beta Distribution-guided aspect-aware graph construction based on external knowledge . they use aspect-related words as the pivots to derive aspect-relevant weights . |
| Outcome: | The proposed approach outperforms the state-of-the-art methods on 6 benchmark datasets. |
UniSumEval: Towards Unified, Fine-grained, Multi-dimensional Summarization Evaluation for LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks for summarization quality evaluation lack diverse input scenarios, focus on narrowly defined dimensions, and struggle with subjective and coarse-grained annotation schemes. |
| Approach: | They propose to use AI to help human annotations and identifie potentially hallucinogenic input texts. |
| Outcome: | The proposed benchmarks improve on existing benchmarks in terms of input diversity, granularity of human annotations, and evaluation dimensions. |
Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis (2022.emnlp-main)
Copied to clipboard
Shuai Fan, Chen Lin, Haonan Li, Zhenghao Lin, Jinsong Su, Hang Zhang, Yeyun Gong, JIan Guo, Nan Duan
| Challenge: | Existing pre-trained language representation models (PLMs) capture sentiment information from word-level while under-considering sentence-level information. |
| Approach: | They propose a Sentiment-aware pre-trained language model with combined Word-level and Sentence-level Pre-training tasks that enhance the PLM’s knowledge about sentiment words. |
| Outcome: | The proposed model achieves state-of-the-art on various sentence-level and aspect-level sentiment classification benchmarks. |
ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection (2026.findings-acl)
Copied to clipboard
Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, YiFan Zhang, Tianlong Xu, Zhendong Chu, Aoxiao Zhong, Kun Wang, Hui Xiong, Philip S. Yu, Xuming Hu, Qingsong Wen
| Challenge: | Current mathematical benchmarks focus on evaluating MLLMs’ problem-solving ability, yet there is a crucial gap in addressing more complex scenarios such as error detection. |
| Approach: | They propose to evaluate multimodal error detection by evaluating two sub-tasks error step identification and error categorization. |
| Outcome: | The proposed task evaluates MLLMs' ability to handle multimodal questions compared to text-only models. |
Active Generalized Category Discovery with Diverse LLM Feedback (2026.eacl-long)
Copied to clipboard
| Challenge: | Generalized Category Discovery (GCD) is a practical and challenging open-world task that aims to recognize both known and novel categories in unlabeled data using limited labeled data from known categories. |
| Approach: | They propose a framework for generalized category discovery that actively learns from diverse and collaborative feedback. |
| Outcome: | The proposed framework improves instance-level contrastive features, generates category descriptions, and aligns uncertain instances with LLM-selected category descriptions. |
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization (2024.naacl-long)
Copied to clipboard
Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, Kathleen McKeown
| Challenge: | Existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of the model’s size. |
| Approach: | They propose to evaluate topic-focused dialogue summarization by using large language models (LLMs) they use human annotations to evaluate factual consistency and explain factually inconsistent sentences. |
| Outcome: | The proposed evaluation benchmark on topic-focused dialogue summarization shows that existing LLMs hallucinate significant amounts of factual errors regardless of the model’s size. |
Exploring the Generalizability of Factual Hallucination Mitigation via Enhancing Precise Knowledge Utilization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to improve LLMs' ability to align their responses with objective facts suffer from poor generalization and trade-offs in other different capabilities. |
| Approach: | They propose to introduce PKUE (Precise Knowledge Utilization Enhancement) which fine-tunes the model on self-generated responses to precise and simple factual questions through preference optimization. |
| Outcome: | The proposed enhancements improve LLM’s ability to precisely leverage its knowledge and improve FactualBench, a comprehensive and precise factual QA dataset containing 181k Chinese data spanning 21 domains. |
TP-RAG: Benchmarking Retrieval-Augmented Large Language Model Agents for Spatiotemporal-Aware Travel Planning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on large language models (LLMs) focus on basic plan validity, but neglect critical aspects such as route efficiency, POI appeal, and real-time adaptability. |
| Approach: | They propose a benchmark for retrieval-augmented, spatiotemporal-aware travel planning that integrates retrieved trajectories with LLMs’ intrinsic reasoning. |
| Outcome: | The proposed framework improves spatial efficiency and POI rationality while challenging universality and robustness due to conflicting references and noisy data. |
Efficient Mind-Map Generation via Sequence-to-Graph and Reinforced Graph Refinement (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate mind-maps from text are difficult to capture the overall semantics of a document. |
| Approach: | They propose an efficient mind-map generation network that converts a document into a graph via sequence-to-graph. |
| Outcome: | The proposed network reduces inference time by thousands of times compared with existing methods and reveals key semantic structures better than plain text. |
CERET: Cost-Effective Extrinsic Refinement for Text Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate incomplete, biased or misleading outputs in their initial attempts. |
| Approach: | They propose a method for refining text generation that takes into account semantic stability, entailment and inter-sample uncertainty measures. |
| Outcome: | The proposed method outperforms self-consistency and self-rerank baselines under various task setups by 1.6% and 3.5% respectively. |
Enhancing Abstractiveness of Summarization Models through Calibrated Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to generate abstractive summarizations are slow and abstractive, but we propose a novel approach to enhance the level of abstractiveness without sacrificing the informativeness of generated summaries. |
| Approach: | They propose a novel approach to enhance the level of abstractiveness without sacrificing the informativeness of generated summaries by exposing diverse pseudo summary with two supervision to the student model. |
| Outcome: | The proposed method outperforms previous methods in abstractive summarization distillation, producing highly abstractive and informative summaries. |
Multi-Label Few-Shot Learning for Aspect Category Detection (2021.acl-long)
Copied to clipboard
| Challenge: | Existing few-shot learning methods focus on single-label predictions, which can not work well for ACD since a sentence may contain multiple aspect categories. |
| Approach: | They propose a few-shot learning method that uses the prototypical network to learn aspects from a set of aspects. |
| Outcome: | The proposed method significantly outperforms baseline methods on three datasets. |
AutoBreach: Universal and Adaptive Jailbreaking with Efficient Wordplay-Guided Optimization via Multi-LLMs (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing jailbreak research exhibits limitations in universality, validity, and efficiency . Existing methods for jailbreaking LLMs have limited validity and effectiveness . |
| Approach: | They propose a black-box approach that uses wordplay-guided mapping rule sampling to create universal adversarial prompts. |
| Outcome: | The proposed method efficiently identifies security vulnerabilities across various LLMs, achieving an average success rate of over 80% with fewer than 10 queries. |
Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders (2024.acl-long)
Copied to clipboard
| Challenge: | Conversational systems often rely on embedding models for intent classification and intent clustering tasks. |
| Approach: | They propose a toolkit that gives a more holistic view of intent embedding models by considering three tasks– (1) intent classification, (2) intent clustering, and (3) a novel triplet task. |
| Outcome: | The proposed model improves on the linguistic dimensions while affecting performance on downstream task metrics. |
Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection (2024.naacl-long)
Copied to clipboard
| Challenge: | Semi-supervised dialogue summarization (SSDS) leverages model-generated summaries to reduce reliance on human-labeled data. |
| Approach: | They propose a scoring approach that encapsulates three primary dimensions of summarization model quality. |
| Outcome: | The proposed method reduces reliance on human-labeled data and improves the performance of summarization models. |
The Subtle Art of Defection: Understanding Uncooperative Behaviors in LLM based Multi-Agent Systems (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing literature on uncooperative behavior degrades collective outcomes and requires more resilient multi-agent systems. |
| Approach: | They propose a game theory-based taxonomy of uncooperative agent behaviors and a structured, multi-stage simulation pipeline that dynamically generates and refines uncooperation behaviors as agents’ states evolve. |
| Outcome: | The proposed framework achieves 96.7% accuracy in generating realistic uncooperative behaviors, validated by human evaluations. |
Understanding and Improving Information Preservation in Prompt Compression for LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models have enabled their successful application to a broad range of tasks. |
| Approach: | They propose a framework that allows for in-depth analysis of prompt compression methods. |
| Outcome: | The proposed framework analyzes state-of-the-art soft and hard compression methods . it shows that some fail to preserve key details from the original prompt, limiting performance on complex tasks. |
Red Teaming Large Reasoning Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) have emerged as a powerful advancement in multi-step reasoning tasks, but they introduce safety and reliability risks, such as CoT-hijacking and prompt-induced inefficiencies. |
| Approach: | They propose a unified benchmark to assess the trustworthiness of Large Reasoning Models. |
| Outcome: | The proposed benchmark evaluates truthfulness, safety and efficiency on 26 models. |
CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs rely on surface-level similarity or manual heuristics to evaluate personalization . Existing evaluation protocols for personalization are lacking sufficient data-driven validation. |
| Approach: | They propose a benchmark to assess personalization by mining CIPDs to quantify individual preferences. |
| Outcome: | The proposed benchmark provides a more comprehensive and discriminative standard than generic metrics. |
Personalized Question Answering with User Profile Generation and Compression (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models are prone to providing “midguy” answers regardless of users’ knowledge background, thereby failing to meet each user’s personalized needs. |
| Approach: | They propose to generate personalized answers with LLMs based on users’ past question-answering records. |
| Outcome: | The proposed method generates personalized answers based on user's past question-answering records. |
Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation (2025.naacl-long)
Copied to clipboard
Mahnaz Koupaee, Jake W. Vincent, Saab Mansour, Igor Shalyminov, Han He, Hwanjun Song, Raphael Shu, Jianfeng He, Yi Nian, Amy Wing-mei Wong, Kyu J. Han, Hang Su
| Challenge: | Existing approaches to evaluate faithfulness of summaries are often fooled by the fluency of the text and struggle with identifying errors. |
| Approach: | They propose an approach to summary faithfulness evaluation where multiple LLM-based agents are assigned initial stances and forced to come up with a reason to justify belief. |
| Outcome: | The proposed approach can identify ambiguities and have even stronger performance on non-ambiguous summaries. |
Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages (2025.acl-long)
Copied to clipboard
Hyangsuk Min, Yuho Lee, Minjeong Ban, Jiaqi Deng, Nicole Hee-Yeon Kim, Taewon Yun, Hang Su, Jason Cai, Hwanjun Song
| Challenge: | Existing evaluation frameworks for text summarization lack domain-specific assessment criteria and are predominantly English-centric. |
| Approach: | They propose a multi-dimensional, multi-domain evaluation of summarization in English and Chinese that incorporates specialized assessment criteria for each domain and leverages a debate system to enhance annotation quality. |
| Outcome: | The proposed evaluation framework provides a multi-dimensional, multi-domain evaluation of summarization in English and Chinese. |
LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Tasks (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing defense methods are insufficient to address in-context reward hacking (ICRH), where LLMs iteratively optimize their behavior to maximize proxy objectives, resulting in harmful side effects. |
| Approach: | They propose a framework that reduces in-context reward hacking (ICRH) through repeated interactions with the environment. |
| Outcome: | The proposed framework reduces ICRH without model fine-tuning while maintaining task performance. |
Learning to Summarize from LLM-generated Feedback (2025.naacl-long)
Copied to clipboard
| Challenge: | Developing effective text summarizers remains a challenge due to issues like unfaithful statements, key information omissions, and verbosity. |
| Approach: | They propose a large-scale dataset containing multi-dimensional feedback on LLM-generated summaries of varying quality across diverse domains to align them with human preferences for faithfulness, completeness, and conciseness. |
| Outcome: | The proposed model outperforms the 10x larger Llama3-70b-instruct in generating human-preferred summaries. |