Papers by Alexander Liu
Codec-SUPERB: An In-Depth Analysis of Sound Codec Models (2024.findings-acl)
Copied to clipboard
Haibin Wu, Ho-Lam Chung, Yi-Cheng Lin, Yuan-Kuei Wu, Xuanjun Chen, Yu-Chi Pai, Hsiu-Hsuan Wang, Kai-Wei Chang, Alexander Liu, Hung-yi Lee
| Challenge: | Researchers have developed a sound codec that can be used as tokenizers for preserving audio data and minimizing data transmission latency. |
| Approach: | They propose to use codec-SUPERB to assess codec models across representative sound applications and signal-level metrics rooted in sound domain knowledge. |
| Outcome: | The proposed codec-SUPERB model is evaluated on selected experimental settings. |
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on text summarization factual consistency are divided into two categories . entailment-based and question answering-based metrics are the most efficient . |
| Approach: | They propose an optimized QA-based metric that improves factual consistency by 14% . they compare entailment-based and QA metrics to find the best fit . |
| Outcome: | The proposed metric outperforms the best performing entailment-based metric on the SummaC factual consistency benchmark. |
FOLIO: Natural Language Reasoning with First-Order Logic (2024.emnlp-main)
Copied to clipboard
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev
| Challenge: | Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity. |
| Approach: | They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models. |
| Outcome: | The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models. |
On Learning to Summarize with Large Language Models as References (2024.naacl-long)
Copied to clipboard
Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu, Dragomir Radev, Arman Cohan
| Challenge: | Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators when compared to reference summary from widely used summarization datasets. |
| Approach: | They propose to use large language models (LLMs) as reference learning settings for smaller text summarization models to investigate whether their performance can be substantially improved. |
| Outcome: | The proposed model outperforms standard supervised fine-tuning and human evaluations while retaining human-level performance. |
Predicting Text Preference Via Structured Comparative Reasoning (2024.acl-long)
Copied to clipboard
Jing Nathan Yan, Tianqi Liu, Justin Chiu, Jiaming Shen, Zhen Qin, Yue Yu, Charumathi Lakshmanan, Yair Kurzion, Alexander Rush, Jialu Liu, Michael Bendersky
| Challenge: | Existing approaches to comparative reasoning rely on pretraining or fine-tuning models at the cost of massive human annotation and computation. |
| Approach: | They propose a model that prompts LLMs to generate structured intermediate comparisons by proposing aspects for comparison, followed by generating textual comparisons under each aspect. |
| Outcome: | The proposed model significantly reduces hallucination and improves consistency across various NLP tasks. |
Are Large Language Models Capable of Generating Human-Level Narratives? (2024.emnlp-main)
Copied to clipboard
Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, Nanyun Peng
| Challenge: | a recent HCI study has pointed to gaps in machine storytelling ability at the global level . authors show that LLMs have less suspense and less tension than human stories . |
| Approach: | They propose a computational framework to analyze narratives through three discourse-level aspects. |
| Outcome: | The proposed framework analyzes narratives through three discourse-level aspects . it shows that LLMs fall short of human abilities in discourse understanding . |
Optimizing Retrieval-Augmented Generation for E-Commerce How-To Assistance (2026.acl-industry)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have significantly improved Retrieval-Augmented Generation (RAG), enabling assistants that can reliably ground responses in external knowledge sources while maintaining high-quality natural language interaction. |
| Approach: | They propose a RAG-based How-To Assistant that groundes responses in a proprietary knowledge base to provide personalized customer support. |
| Outcome: | The proposed assistant can ground responses in a proprietary knowledge base while maintaining high-quality natural language interaction. |
Cross-Modal Discrete Representation Learning (2022.acl-long)
Copied to clipboard
| Challenge: | a new framework for learning representations from multimodal data is proposed . the proposed framework uses discretized embedding vectors to capture finer levels of granularity . |
| Approach: | They propose a self-supervised representation learning framework that captures finer levels of granularity across different modalities. |
| Outcome: | The proposed representation can capture finer levels of granularity across different modalities . it can be used on cross-modal retrieval tasks without direct supervision . |
End-to-End Evaluation for Low-Latency Simultaneous Speech Translation (2023.emnlp-demo)
Copied to clipboard
Christian Huber, Tu Anh Dinh, Carlos Mullov, Ngoc-Quan Pham, Thai Binh Nguyen, Fabian Retkowski, Stefan Constantin, Enes Ugan, Danni Liu, Zhaolin Li, Sai Koneru, Jan Niehues, Alexander Waibel
| Challenge: | a framework to evaluate low-latency speech translations is currently only limited to specific aspects and is not able to compare different approaches. |
| Approach: | They propose a framework to perform and evaluate low-latency speech translation in realistic conditions. |
| Outcome: | The proposed framework evaluates various aspects of low-latency speech translation under realistic conditions. |
Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023.emnlp-main)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Compared to neural systems, automatic metrics should be interpretable and provide intuitive insights into system performance and output quality. |
| Approach: | They propose to use a two-stage evaluation pipeline to extract basic information units from one text sequence and check the extracted units in another sequence. |
| Outcome: | The proposed metrics can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability. |
Learning to Extract Structured Entities Using Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) play a pivotal role in extracting structured information from unstructured text. |
| Approach: | They propose to reformulate the task to be entity-centric, enabling the use of diverse metrics that can provide more insights from various perspectives. |
| Outcome: | The proposed model outperforms baselines and human evaluations on the extracted entities. |
ReIFE: Re-evaluating Instruction-Following Evaluation (2025.naacl-long)
Copied to clipboard
Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan
| Challenge: | Existing evaluations of large language models (LLMs) for instruction following are incomplete. |
| Approach: | They propose to use 25 base LLMs and 15 recently proposed evaluation protocols to evaluate instruction following on 4 human-annotated datasets. |
| Outcome: | The proposed evaluations identify the best-performing base LLMs and evaluation protocols with a high degree of robustness. |
Semantic Novelty Detection and Characterization in Factual Text Involving Named Entities (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing topic-based novelty detection methods do not perform semantic reasoning involving relations between named entities in text and their background knowledge. |
| Approach: | They propose a model to detect whether a text is novel or not . they propose to use a factual text to characterize novelty. |
| Outcome: | The proposed model outperforms 10 baselines by large margins on the novelty detection task. |
Fair Abstractive Summarization of Diverse Perspectives (2024.naacl-long)
Copied to clipboard
Yusen Zhang, Nan Zhang, Yixin Liu, Alexander Fabbri, Junru Liu, Ryo Kamoi, Xiaoxin Lu, Caiming Xiong, Jieyu Zhao, Dragomir Radev, Kathleen McKeown, Rui Zhang
| Challenge: | Existing work on summarization metrics and large language models has not explored fair abstractive summarizing. |
| Approach: | They propose four reference-free automatic metrics to measure the differences between target and source perspectives. |
| Outcome: | The proposed methods alleviate fair abstractive summarization on user-generated data. |
A Sliding-Window Approach to Automatic Creation of Meeting Minutes (2021.naacl-srw)
Copied to clipboard
| Challenge: | Existing methods to extract utterances and keyphrases from transcripts are lacking in meeting minutes. |
| Approach: | They propose a sliding-window approach to automatic generation of meeting minutes . they use a neural abstractive abstractive to navigate through the raw transcript . |
| Outcome: | The proposed approach is evaluated on natural transcripts and two versions of automatic transcripts. |
Exploring Neural Models for Query-Focused Summarization (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work in Query-focused summarization lacks a comprehensive study of the broad space of applicable modeling methods. |
| Approach: | They propose to explore two general classes of methods for Query-focused summarization: extractive-abstractive solutions and end-to-end models. |
| Outcome: | The proposed models achieve state-of-the-art on the QMSum dataset, with a margin of 3.38 ROUGE-1, 3.72 ROUGe2 and 3.28 ROUGEL-L. |
Semantic Novelty Detection in Natural Language Descriptions (2021.emnlp-main)
Copied to clipboard
Nianzu Ma, Alexander Politowicz, Sahisnu Mazumder, Jiahua Chen, Bing Liu, Eric Robertson, Scott Grigsby
| Challenge: | Existing novelty detection algorithms are coarse-grained, working at the document or topic level. |
| Approach: | They propose to use a fine-grained semantic novelty detection problem to solve a novel novel scene problem. |
| Outcome: | The proposed model outperforms baseline models on the proposed task by large margins. |
Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024.findings-naacl)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan
| Challenge: | Recent studies have found that large language models (LLMs) can achieve state-of-the-art performance on generic summarization benchmarks, but their performance on more complex summarizing task settings is less studied. |
| Approach: | They benchmark large language models on instruction controllable text summarization . they use 4 evaluation protocols and 11 LLMs to evaluate their performance . |
| Outcome: | The proposed model performs well on instruction controllable text summarization tasks with 4 evaluation protocols and 11 LLMs. |
SHIELD: LLM-Driven Schema Induction for Predictive Analytics in EV Battery Supply Chain Disruptions (2024.emnlp-industry)
Copied to clipboard
Zhi-Qi Cheng, Yifei Dong, Aike Shi, Wei Liu, Yuzhi Hu, Jason O’Connor, Alexander Hauptmann, Kate Whitefoot
| Challenge: | EV battery supply chain is vulnerable to disruptions caused by natural disasters and geopolitical tensions. |
| Approach: | They propose a system integrating Large Language Models with domain expertise for EV supply chain risk assessment. |
| Outcome: | Evaluated on 12,070 paragraphs from 365 sources (2022-2023), SHIELD outperforms baseline GCNs and LLM+prompt methods in disruption prediction. |
Transfer Learning for Entity Recognition of Novel Classes (C18-1)
Copied to clipboard
| Challenge: | Existing approaches to entity recognition are based on class labels in source and target domains, and many NER corpora only annotate a small number of categories. |
| Approach: | They replicate and extend several past studies on transfer learning for entity recognition. |
| Outcome: | The proposed methods perform better when there is more labeled target data. |
How Domain Terminology Affects Meeting Summarization Performance (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to summarize meetings using domain terminology are understudied . jargon terms are used to identify salient utterances from transcripts . |
| Approach: | They propose to use jargon terms to identify salient utterances from transcribed meetings to generate meeting minutes. |
| Outcome: | The proposed system generates minutes from transcribed meetings by identifying salient utterances from transcripts. |
Improving the Distributional Alignment of LLMs using Supervision (2026.acl-long)
Copied to clipboard
Gauri Kambhatla, Sanjana Gautam, Angela Zhang, Alexander Liu, Ravi Srinivasan, Junyi Jessy Li, Matthew Lease
| Challenge: | Existing work to evaluate LLMs' alignment with human values and opinions has a key shortcoming. |
| Approach: | They propose to add supervision to LLMs to improve alignment with diverse populations . they find that supervision improves alignment across public health, public opinion, values and beliefs . |
| Outcome: | The proposed method improves the alignment of LLMs with diverse populations on subjective questions. |
Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs (2026.acl-long)
Copied to clipboard
Nikita Afonin, Nikita Andriianov, Vahagn Hovhannisyan, Nikhil Bageshpura, Kyle Liu, Kevin Zhu, Sunishchal Dev, Ashwinee Panda, Oleg Rogov, Elena Tutubalina, Alexander Panchenko, Mikhail Seleznyov
| Challenge: | Recent studies have documented emergent misalignment in language models adapted on narrow examples . Emergent misalignement occurs when models are trained on narrow set of misallocated examples resulting in harmful or misleading responses . |
| Approach: | They propose to explain in-context EM as conflict between safety objectives and context-following behavior. |
| Outcome: | The proposed model is adapted on 16 in-context examples and produces misaligned responses to benign queries. |
We’re Afraid Language Models Aren’t Modeling Ambiguity (2023.emnlp-main)
Copied to clipboard
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, Yejin Choi
| Challenge: | Ambiguity is an intrinsic feature of natural language, allowing us to anticipate misunderstandings and revise our interpretations as listeners. |
| Approach: | They use AmbiEnt to capture ambiguity in a sentence and analyze it to evaluate pretrained LMs. |
| Outcome: | The proposed model can flag political claims in the wild that are misleading due to ambiguity. |
SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading (2024.emnlp-main)
Copied to clipboard
Tu Dinh, Carlos Mullov, Leonard Bärmann, Zhaolin Li, Danni Liu, Simon Reiß, Jueun Lee, Nathan Lerzer, Jianfeng Gao, Fabian Peller-Konrad, Tobias Röddiger, Alexander Waibel, Tamim Asfour, Michael Beigl, Rainer Stiefelhagen, Carsten Dachsbacher, Klemens Böhm, Jan Niehues
| Challenge: | Large Language Models (LLMs) are rapidly developing and are becoming more and more useful in scientific tasks. |
| Approach: | They propose to use LLM-as-a-judge to grade LLMs on SciEx to assess their ability on scientific tasks. |
| Outcome: | The proposed benchmarks show that the LLMs perform decently on free-form exams, achieving 0.948 Pearson correlation with expert grading. |
Dial HEALTHDIAL for Advice: A Multilingual and Multi-Parallel Spoken Dialogue Dataset for Knowledge-Grounded Information Seeking (2026.findings-acl)
Copied to clipboard
Songbo Hu, Yinhong Liu, Ej Zhou, Evgeniia Razumovskaia, Xiaobin Wang, Alexander Fraser, Ivan Vulić, Anna Korhonen
| Challenge: | Creating spoken dialogue datasets is methodologically challenging due to the personally identifiable nature of speech signals. |
| Approach: | They propose a large-scale, multilingual, and multi-parallel dataset for developing and evaluating retrieval-augmented generation-based spoken dialogue systems. |
| Outcome: | The proposed dataset includes 6,000 information-seeking dialogues and 163 hours of user speech recorded from native speakers of four official WHO languages. |