Papers by Michael Liu
Want To Reduce Labeling Cost? GPT-3 Can Help (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Data annotation is labor-intensive and time-consuming for many NLP tasks. |
| Approach: | They propose to use GPT-3 to train models which are deployed for inference . they propose to combine pseudo labels from GPT3 with human labels . |
| Outcome: | The proposed method can be generalizable to many practical applications. |
ParaTag: A Dataset of Paraphrase Tagging for Fine-Grained Labels, NLG Evaluation, and Data Augmentation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets only annotate a binary label for each sentence pair. Existing models only annnotate binary labels for each phrase pair. |
| Approach: | They propose a novel binary paraphrase classification task that annotates the degree of paraphrase between sentences and a new annotation schema that labels the minimum spans of tokens in a sentence that don't have the corresponding paraphrases in the other sentence. |
| Outcome: | The proposed dataset can be used to train an automatic scorer for language generation evaluation. |
Sequential LLM Framework for Fashion Recommendation (2024.emnlp-industry)
Copied to clipboard
Han Liu, Xianfeng Tang, Tianlang Chen, Jiapeng Liu, Indu Indu, Henry Zou, Peng Dai, Roberto Galan, Michael Porter, Dongmei Jia, Ning Zhang, Lian Xiong
| Challenge: | Existing fashion recommendation systems struggle with the unique challenges of the fashion domain. |
| Approach: | They propose a sequential fashion recommendation framework that leverages a pre-trained large language model enhanced with recommendation-specific prompts. |
| Outcome: | The proposed framework significantly improves fashion recommendation performance on Amazon fashion. |
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions (2025.emnlp-main)
Copied to clipboard
Emmy Liu, Amanda Bertsch, Lintang Sutawika, Lindia Tjuatja, Patrick Fernandes, Lara Marinov, Michael Chen, Shreya Singhal, Carolin Lawrence, Aditi Raghunathan, Kiril Gashteovski, Graham Neubig
| Challenge: | Language model performance is largely dependent on pretraining decisions, but scaling laws based on only these two aspects do not always explain downstream task performance. |
| Approach: | They meta-analyze 92 open-source pretrained models to quantify their impact on performance. |
| Outcome: | The framework lays a foundation for more systematic investigation of how model development choices shape final capabilities. |
Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Discourse connectives are words or phrases that signal the presence of a discourse relation. |
| Approach: | They propose a model that generates discourse connectives between arguments and predicts discourse relations based on the generated connectives. |
| Outcome: | The proposed model outperforms baselines on three datasets and is highly accurate. |
Exploring End-to-End Differentiable Natural Logic Modeling (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to integrate natural logic with neural networks are brittle and prone to fail in the presence of noise and uncertainty. |
| Approach: | They propose to integrate natural logic with neural networks to create differentiable models that integrate natural reasoning with subsymbolic vector representations and neural components. |
| Outcome: | The proposed model can model monotonicity-based reasoning, compared to baseline models without inductive bias. |
End-to-End Segmentation-based News Summarization (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing summarization systems only provide one genetic summary of the whole article, making it difficult for users to navigate the reading. |
| Approach: | They propose a task of segmenting a news article into multiple sections and generating the corresponding summary to each section. |
| Outcome: | The proposed model outperforms state-of-the-art models on a 27k news article dataset . it can jointly segment a document and produce the summary for each section . |
What Causes the Failure of Explicit to Implicit Discourse Relation Recognition? (2024.naacl-long)
Copied to clipboard
| Challenge: | Prior work claimed that explicit classifiers perform poorly in implicit scenarios . a label shift occurs after connectives are removed, but no empirical evidence supports this claim . |
| Approach: | They propose to prove that the discourse relations expressed by some explicit instances will change when connectives disappear. |
| Outcome: | The proposed methods outperform strong baselines on PDTB 2.0, PDTT 3.0, and the GUM dataset. |
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization (2023.acl-long)
Copied to clipboard
Pengcheng He, Baolin Peng, Song Wang, Yang Liu, Ruochen Xu, Hany Hassan, Yu Shi, Chenguang Zhu, Wayne Xiong, Michael Zeng, Jianfeng Gao, Xuedong Huang
| Challenge: | Z-Code++ is a pre-trained language model optimized for abstractive text summarization. |
| Approach: | They propose a pre-trained language model optimized for abstractive text summarization that uses a two-phase pre-training technique to improve model's performance. |
| Outcome: | The proposed model outperforms the competing models on low-resource summarization tasks in zero-shot and few-shot settings. |
A Survey of NLP Progress in Sino-Tibetan Low-Resource Languages (2025.naacl-long)
Copied to clipboard
| Challenge: | Despite the increasing effort in including more low-resource languages in NLP/CL development, most of the world’s languages are still absent. |
| Approach: | They propose to include low-resource languages in NLP/CL research as more resources are poured into the development of data-driven models. |
| Outcome: | The proposed language family is a low-resource language family with a small number of native speakers and government support. |
PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning (2026.acl-long)
Copied to clipboard
Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph IV, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego A. Mares Buendia, Pavit Singh, Michael Liu, Subodh Chawla, Peter Cline, Lucy Ogaz, Ernesto Gabriel Hernández Montoya, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu, Yunzhong He
| Challenge: | Frontier models often lack a view of performance on open-ended, economically consequential tasks in high-stakes professional domains where practical returns matter most. |
| Approach: | They introduce a professional reasoning benchmark that recruits 182 qualified professionals to contribute questions inspired by their workflows. |
| Outcome: | The proposed model outperforms other models in 114 countries and 47 US jurisdictions on hard subsets. |
InstructCoder: Instruction Tuning Large Language Models for Code Editing (2024.acl-srw)
Copied to clipboard
| Challenge: | InstructCoder is the first instruction-tuning dataset designed to adapt LLMs for general-purpose code editing. |
| Approach: | They propose to use Large Language Models to edit code based on user instructions . they use a dataset to adapt LLMs to general-purpose code editing . |
| Outcome: | The proposed model can significantly improve code editing performance compared to proprietary models . the proposed model is based on a human-written execution-based benchmark . |
TaKG: A New Dataset for Paragraph-level Table-to-Text Generation Enhanced with Knowledge Graphs (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing table-to-text generation benchmarks have some limitations, such as E2E and ToTTo focusing on singlesentence generation tasks. |
| Approach: | They propose a new table-to-text generation dataset called TaKG that uses a set of knowledge graphs to enhance table input. |
| Outcome: | The proposed model outperforms existing models for short-text generation tasks and shows reliable performance on long-text generated across a variety of metrics. |
REIC: RAG-Enhanced Intent Classification at Scale (2025.emnlp-industry)
Copied to clipboard
Ziji Zhang, Michael Yang, Zhiyu Chen, Yingying Zhuang, Shu-Ting Pi, Qun Liu, Rajashekar Maragoud, Vy Nguyen, Anurag Beniwal
| Challenge: | Accurate intent classification is critical for efficient routing in customer service . however, as companies expand their product lines, intent classification faces scalability challenges . |
| Approach: | They propose a retrieval-augmented generation Enhanced Intent Classification approach which leverages retrieval augmented generation to integrate relevant knowledge into a model. |
| Outcome: | The proposed approach outperforms fine-tuning, zero-shot, and few-shot methods on real-world datasets. |
Agent Laboratory: Using LLM Agents as Research Assistants (2025.findings-emnlp)
Copied to clipboard
Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, Emad Barsoum
| Challenge: | Agent Laboratory is an autonomous LLM-based framework that can complete the entire research process. |
| Approach: | Agent Laboratory is an autonomous LLM-based framework that can complete the entire research process. |
| Outcome: | Agent Laboratory is an autonomous LLM-based framework that can complete the entire research process. |
OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens (2025.acl-demo)
Copied to clipboard
Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, Yen-Sung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Yu-Yen Cheng, Karen Farley, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, Jesse Dodge
| Challenge: | tracing language models' outputs back to training data is a problem because they are trained on text corpora with trillions of tokens . existing methods for tracers have not been scaled to work within this multi-trillion-token setting . |
| Approach: | They propose a system that traces language models' outputs verbatim back to training data . OLMOTRACE retrieves documents from the model's training data that contain exact matches . |
| Outcome: | The proposed system can find verbatim matches between LM output and training data . it can be used to explore fact checking, hallucination, and creativity of language models . |
JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been used to mitigate misuse and to align with human values. |
| Approach: | They propose to use large-scale evaluations of various jailbreak attacks to identify key patterns and test them under eight advanced defenses. |
| Outcome: | The proposed attacks achieve high success rates but are easy to mitigate by defenses. |
DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling (2020.findings-emnlp)
Copied to clipboard
Jiecao Chen, Liu Yang, Karthik Raman, Michael Bendersky, Jung-Jung Yeh, Yun Zhou, Marc Najork, Danyang Cai, Ehsan Emadzadeh
| Challenge: | Existing knowledge distillation models are not optimized for dealing with pairs (or tuples) of texts. |
| Approach: | They propose a framework for distilling fast and accurate models on text pair tasks using a scalable end-to-end training strategy. |
| Outcome: | Empirical studies on academic and real-world e-commerce benchmarks show the proposed framework can achieve speedups of over 350x and minimal quality drop relative to the cross-attention teacher BERT model. |
Evaluating the Prompt Steerability of Large Language Models (2025.naacl-long)
Copied to clipboard
Erik Miehling, Michael Desmond, Karthikeyan Natesan Ramamurthy, Elizabeth M. Daly, Kush R. Varshney, Eitan Farchi, Pierre Dognin, Jesus Rios, Djallel Bouneffouf, Miao Liu, Prasanna Sattigeri
| Challenge: | a primary question underlying alignment research is: whose views are we aligning to? |
| Approach: | They propose to evaluate the steerability of model personas as a function of prompting by defining a benchmark and inspecting how these indices change as if steering effort is a factor. |
| Outcome: | The proposed benchmark reveals that the steerability of many current models is limited due to skew in baseline behavior and an asymmetry in their steerability across many persona dimensions. |
Fusing Context Into Knowledge Graph for Commonsense Question Answering (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to combine language modeling and knowledge graphs (KG) lack the context to provide a more precise understanding of the concepts. |
| Approach: | They propose to use external entity descriptions to provide contextual information for commonsense question answering models. |
| Outcome: | The proposed model achieves state-of-the-art among non-generative models in OpenBookQA and is the first of its kind in the field. |
SConE: Simplified Cone Embeddings with Symbolic Operators for Complex Logical Queries (2023.findings-acl)
Copied to clipboard
| Challenge: | Current geometric-based methods depend on the neural approach to model FOL operators . empirical evidence for explainability is challenging . |
| Approach: | They propose to model conjunction operators using a symbolic modeling approach . they propose to emphasize the essential role of relation projection operator . |
| Outcome: | The proposed method improves answering complex logical queries over previous models. |
MaintIE: A Fine-Grained Annotation Schema and Benchmark for Information Extraction from Maintenance Short Texts (2024.lrec-main)
Copied to clipboard
| Challenge: | Maintenance short texts (MSTs) provide crucial insights into the state and maintenance activities of machines, infrastructure, and other engineered assets. |
| Approach: | They propose a multi-level fine-grained annotation scheme for entity recognition and relation extraction that includes 5 top-level classes and 6 relations tailored to MSTs. |
| Outcome: | The proposed scheme provides high-quality, fine-grained annotations and a coarse-grain corpus of 7,000 texts. |
Predicting Text Preference Via Structured Comparative Reasoning (2024.acl-long)
Copied to clipboard
Jing Nathan Yan, Tianqi Liu, Justin Chiu, Jiaming Shen, Zhen Qin, Yue Yu, Charumathi Lakshmanan, Yair Kurzion, Alexander Rush, Jialu Liu, Michael Bendersky
| Challenge: | Existing approaches to comparative reasoning rely on pretraining or fine-tuning models at the cost of massive human annotation and computation. |
| Approach: | They propose a model that prompts LLMs to generate structured intermediate comparisons by proposing aspects for comparison, followed by generating textual comparisons under each aspect. |
| Outcome: | The proposed model significantly reduces hallucination and improves consistency across various NLP tasks. |
Defending Against Social Engineering Attacks in the Age of LLMs (2024.emnlp-main)
Copied to clipboard
Lin Ai, Tharindu Kumarage, Amrita Bhattacharjee, Zizhou Liu, Zheng Hui, Michael Davinroy, James Cook, Laura Cassani, Kirill Trapeznikov, Matthias Kirchner, Arslan Basharat, Anthony Hoogs, Joshua Garland, Huan Liu, Julia Hirschberg
| Challenge: | Existing research has developed frameworks to understand human-to-human CSE attacks. |
| Approach: | They propose a modular defense pipeline that improves detection at both the message and conversation levels. |
| Outcome: | The proposed model can be exploited to facilitate chat-based social engineering attacks and generate high-quality CSE content, but their detection capabilities are suboptimal, leading to increased operational costs for defense. |
Are We in the AI-Generated Text World Already? Quantifying and Monitoring AIGT on Social Media (2025.acl-long)
Copied to clipboard
| Challenge: | Social media platforms are experiencing a growing presence of AI-Generated Texts (AIGTs) however, the misuse of AIGTs could have profound implications for public opinion . |
| Approach: | They collect a dataset with 2.4M posts from 3 major social media platforms . they then construct a diverse dataset to train and evaluate AIGT detectors . |
| Outcome: | The proposed dataset analyzes 2.4M posts from 3 major social media platforms from 2022 to 2024 . it finds that Medium and Quora show marked increases in AAR . |
Advancing Adversarial Suffix Transfer Learning on Aligned Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent efforts have identified adversarial suffixes capable of jailbreaking LLMs . however, GCG struggles with computational inefficiency, limiting further studies . |
| Approach: | They propose a two-stage transfer learning framework which decouples the search process into behavior-agnostic pre-searching and behavior-relevant post-search. |
| Outcome: | The proposed approach outperforms baseline on Llama2-chat-7b with ASRs of 43.9 (+ 22.2) and 39.0 (+ 19.5) on valid and test sets. |
Retrieval Enhanced Model for Commonsense Generation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing frameworks for commonsense generation are lacking for pre-trained models. |
| Approach: | They propose a framework that uses concept matching to retrieve prototype sentences and trainable sentence retriever to enhance pre-training and fine-tuning. |
| Outcome: | The proposed framework achieves state-of-the-art on the large-scale Common-Gen benchmark. |
Dialogue Generation on Infrequent Sentence Functions via Structured Meta-Learning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Sentence function is an important linguistic feature indicating the communicative purpose of a sentence in a conversation. |
| Approach: | They propose a structured meta-learning approach for dialogue generation on infrequent sentence functions. |
| Outcome: | The proposed approach improves informativeness and relevance of dialogue generation on infrequent sentence functions while preserving knowledge generalization for similar sentence functions. |
MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing datasets for dialogue summarization are limited to their small sizes and are built from a narrow domain. |
| Approach: | They propose a large-scale media interview dataset consisting of 463.6K transcripts with abstractive summaries. |
| Outcome: | The proposed dataset is larger and contains multi-party conversations from multiple domains. |
BinaryBERT: Pushing the Limit of BERT Quantization (2021.acl-long)
Copied to clipboard
| Challenge: | Recent pre-trained language models have achieved remarkable performance improvement in various tasks, but the improvement generally comes at the cost of increasing model size and computation. |
| Approach: | They propose a binary quantization technique which initializes binaryBERT by splitting from a ternary network. |
| Outcome: | The proposed model achieves state-of-the-art performance on the GLUE and SQUAD benchmarks while being 24x smaller. |
Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets (2021.findings-emnlp)
Copied to clipboard
| Challenge: | despite high accuracy, modern neural networks can still suffer from severe miscalibration. |
| Approach: | They propose to use tag frequency grouping to measure calibration error in different frequency bands to reduce error. |
| Outcome: | The proposed techniques reduce calibration error across the marginal distribution for two existing sequence taggers. |
RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios (2022.naacl-demo)
Copied to clipboard
Xinya Du, Zixuan Zhang, Sha Li, Pengfei Yu, Hongwei Wang, Tuan Lai, Xudong Lin, Ziqi Wang, Iris Liu, Ben Zhou, Haoyang Wen, Manling Li, Darryl Hannan, Jie Lei, Hyounghun Kim, Rotem Dror, Haoyu Wang, Michael Regan, Qi Zeng, Qing Lyu, Charles Yu, Carl Edwards, Xiaomeng Jin, Yizhu Jiao, Ghazaleh Kazeminejad, Zhenhailong Wang, Chris Callison-Burch, Mohit Bansal, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Martha Palmer, Heng Ji
| Challenge: | Existing methods for event prediction are incomplete and noisy. |
| Approach: | They propose to use news-related event schemas to extract newsworthy events . they build a demo website and include a video demonstrating the framework . |
| Outcome: | The proposed framework can be applied to a wide variety of newsworthy scenarios. |
Lightweight Transformers for Conversational AI (2022.naacl-industry)
Copied to clipboard
| Challenge: | Commercial dialogue systems typically require a small footprint and fast execution time, but recent trends are in the other direction, resulting in difficulties in model deployment. |
| Approach: | They build Transformer-based Language Models from scratch on large corpora of conversational data and compare their performance against BERT and other strong baselines on dialogue probing tasks. |
| Outcome: | The proposed model outperforms existing models on dialogue probing tasks and can be fine-tuned on a single consumer GPU card. |
Multilingual Fine-Grained News Headline Hallucination Detection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models to generate news headlines often suffer from the "hallucination" problem, where the produced headline does not fully align with the source article's content. |
| Approach: | They propose to use a multilingual, fine-grained dataset to detect news headlines in 5 languages using supervised fine-tuning techniques and coarse-to-fine prompting to boost the few-shot detection performance. |
| Outcome: | The proposed methods boost the few-shot hallucination detection performance in terms of the example-F1 metric. |
LLM-Guided Semantic Bootstrapping for Interpretable Text Classification with Tsetlin Machines (2026.findings-acl)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) provide strong semantic representations but are costly and opaque. |
| Approach: | They propose a framework that transfers pretrained language models into symbolic form and integrates them into symbolic models. |
| Outcome: | The proposed framework improves interpretability and accuracy across multiple text classification tasks while remaining fully symbolic and efficient. |
Hyperbolic Geometry is Not Necessary: Lightweight Euclidean-Based Models for Low-Dimensional Knowledge Graph Embeddings (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent knowledge graph embedding models based on hyperbolic geometry are complicated than Euclidean operations. |
| Approach: | They propose to use hyperbolic geometry to generate high-fidelity and parsimonious representations of hierarchical patterns in knowledge graphs. |
| Outcome: | The proposed models achieve state-of-the-art performance on two widely-used datasets and cost less than RotH. |
LexiClean: An annotation tool for rapid multi-task lexical normalisation (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Lexical normalisation is the task of identifying and normalising non-canonical tokens (e.g. erroneous spelling, acronyms, etc.) in noisy, non-standard, corpora. |
| Approach: | They propose to use LexiClean to annotate multiple tasks in noisy corpora using in situ token modification and annotation that can be rapidly applied corpus wide. |
| Outcome: | The proposed tool can be rapidly applied corpus wide and can identify and normalise noisy, non-standard, and domain specific corpora. |
Abstract Meaning Representation-Based Logic-Driven Data Augmentation for Logical Reasoning (2024.findings-acl)
Copied to clipboard
Qiming Bao, Alex Peng, Zhenyun Deng, Wanjun Zhong, Gael Gendron, Timothy Pistotti, Neset Tan, Nathan Young, Yang Chen, Yonghua Zhu, Paul Denny, Michael Witbrock, Jiamou Liu
| Challenge: | Empirical evidence shows that our proposed method improves performance across seven downstream tasks. |
| Approach: | They propose a logic-driven data augmentation approach that converts text into AMR graphs and converts them back into text to create augmented data. |
| Outcome: | The proposed method leads on the ReClor leaderboard and improves on seven downstream tasks. |
UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | a new benchmark summarization model is being developed to train few-shot summarizers . a large number of summarizing tasks are required to perform well in heterogeneous datasets. |
| Approach: | They propose a few-shot summarization model pre-trained with multiple summarizing tasks . they propose 'uniSumm' to be prefix-tuned to excel at any few-shot summarisation task . |
| Outcome: | The proposed model outperforms baseline models under automatic and human evaluations and achieves comparable results in human evaluation. |
ACE: A LLM-based Negotiation Coaching System (2024.emnlp-main)
Copied to clipboard
| Challenge: | The rapid progress of LLMs has led to the development of more sophisticated AI tutoring systems. |
| Approach: | They develop an LLM-based assistant for coaching negotiation that provides users with targeted feedback for improvement. |
| Outcome: | The proposed system improves negotiation performance significantly compared to a system that doesn’t provide feedback and one which uses an alternative method. |
ModSCAN: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalities (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models have been widely used but stereotypical biases are unexplored. |
| Approach: | They propose a framework to SCAN stereotypical bias within large vision-language models . they examine stereotype biases with respect to gender and race in three scenarios . |
| Outcome: | The proposed framework can reduce stereotypical biases in large vision-language models . the currently popular models show significant stereotype biase . |
MACSum: Controllable Summarization with Mixed Attributes (2023.tacl-1)
Copied to clipboard
Yusen Zhang, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, Rui Zhang
| Challenge: | Existing work on controllable summarization with mixed attributes lacks designated annotations. |
| Approach: | They propose a human-annotated summarization benchmark for controllable summarizing with mixed attributes based on news and dialogue sources . |
| Outcome: | The proposed dataset contains human-annotated summarization datasets with mixed attributes . hard prompt models yield the best performance on most metrics and human evaluations . mixed-attribute control is still challenging for summarizing tasks . |
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)
Copied to clipboard
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
| Challenge: | Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. |
| Approach: | They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations. |
| Outcome: | The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work. |
The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training (2026.findings-acl)
Copied to clipboard
Rui Zhang, Hongwei Li, Yun Shen, Xinyue Shen, Wenbo Jiang, Guowen Xu, Yang Liu, Michael Backes, Yang Zhang
| Challenge: | Misaligned large language models can magnify harm by exploiting them to undermine safety . et al., 2022b; Bai e.t., 2023): misalignment, realignment and model-specific resistance are important . |
| Approach: | They evaluate four methods to identify a mechanism asymmetry between attack and defense . they find that ORPO is most effective for misalignment, but DPO excels in realignment . |
| Outcome: | The proposed methods show a mechanism asymmetry between attack and defense . the proposed methods excel in realignment, but at the expense of model utility . |
PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs (2024.findings-acl)
Copied to clipboard
Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Haorui Wang, Zhen Qin, Feng Han, Jialu Liu, Simon Baumgartner, Michael Bendersky, Chao Zhang
| Challenge: | Knowledge distillation (KD) is a technique for transferring expertise from large teacher models to compact student models with reduced memory footprints and inference costs. |
| Approach: | They propose to transfer knowledge from large teacher models to compact student models by exploiting teacher-student capacity discrepancies to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. |
| Outcome: | The proposed framework exploits teacher-student capacity discrepancy to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. |
Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting (2023.findings-emnlp)
Copied to clipboard
Benjamin Yan, Ruochen Liu, David Kuo, Subathra Adithan, Eduardo Reis, Stephen Kwak, Vasantha Venugopal, Chloe O’Connell, Agustina Saenz, Pranav Rajpurkar, Michael Moor
| Challenge: | Existing methods for generating reports from medical images conflate the content of the report with its style, which can lead to inaccurate reports. |
| Approach: | They propose a two-step approach to generate radiology reports from medical images using large language models and a graph representation of reports. |
| Outcome: | The proposed approach improves the performance of human evaluations with clinical raters. |
AlphaEdit+: Model Editing in the Presence of Conflicting and Inconsistent Knowledge (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for knowledge editing struggle with knowledge conflicts and inconsistencies. |
| Approach: | They propose a new method for knowledge editing that relaxes null-space constraints and introduces a weighting scheme to mitigate conflicts between new and historical knowledge. |
| Outcome: | The proposed method outperforms existing methods on challenging datasets and outperformed existing methods. |
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2024.findings-naacl)
Copied to clipboard
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, Michael Bendersky
| Challenge: | Existing methods to rank documents using large language models do not understand these challenging ranking formulations. |
| Approach: | They propose to use Pairwise Ranking Prompting to improve ranking performance . they propose to outperform fine-tuned baseline rankers on benchmark datasets . |
| Outcome: | The proposed technique outperforms supervised baselines on benchmark datasets and outperformed other LLM-based solutions by over 10% on average. |
Analyzing the Role of Semantic Representations in the Era of Large Language Models (2024.naacl-long)
Copied to clipboard
Zhijing Jin, Yuen Chen, Fernando Gonzalez Adauto, Jiarui Liu, Jiayi Zhang, Julian Michael, Bernhard Schölkopf, Mona Diab
| Challenge: | Existing studies show the benefits of semantic representations in NLP tasks . Existing work using AMR is concerned with trainable models . |
| Approach: | They propose an AMR-driven chain-of-thought prompting method that uses AMR . they propose to use it to predict which input examples AMR may help or hurt on . |
| Outcome: | The proposed method hurts performance more than it helps on five different tasks. |
CylE: Cylinder Embeddings for Multi-hop Reasoning over Knowledge Graphs (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing geometric-based models cannot handle the logical negation operation . Existing models using cones embeddings are limited to representing queries by two-dimensional shapes . Empirical results show that the performance of multi-hop reasoning task using CylE significantly increases over state-of-the-art geometric- based models for queries without negation. |
| Approach: | They propose a geometric-based model based on three-dimensional shapes with unbounded cylinder embeddings that can handle a complete set of first-order logic operations. |
| Outcome: | Empirical results show that CylE outperforms state-of-the-art models for queries without negation. |
AdaPrompt: Adaptive Model Training for Prompt-based NLP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt-based learning can tackle zero-shot and few-shot NLP tasks . authors propose a method that makes use of pre-trained language models . |
| Approach: | They propose to map NLP tasks into natural language prompts, which are then filled by pre-trained language models. |
| Outcome: | The proposed method outperforms standard prompt-based methods in few-shot settings. |
UniMorph 4.0: Universal Morphology (2022.lrec-1)
Copied to clipboard
Khuyagbaatar Batsuren, Omer Goldman, Salam Khalifa, Nizar Habash, Witold Kieraś, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yustinus Ghanggo Ate, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel El-Khaissi, Tiago Pimentel, Michael Gasser, William Abbott Lane, Mohit Raj, Matt Coler, Jaime Rafael Montoya Samame, Delio Siticonatzi Camaiteri, Esaú Zumaeta Rojas, Didier López Francis, Arturo Oncevay, Juan López Bautista, Gema Celeste Silva Villegas, Lucas Torroba Hennigen, Adam Ek, David Guriel, Peter Dirix, Jean-Philippe Bernardy, Andrey Scherbakov, Aziyana Bayyr-ool, Antonios Anastasopoulos, Roberto Zariquiey, Karina Sheifer, Sofya Ganieva, Hilaria Cruz, Ritván Karahóǧa, Stella Markantonatou, George Pavlidis, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Candy Angulo, Jatayu Baxi, Andrew Krizhanovsky, Natalia Krizhanovskaya, Elizabeth Salesky, Clara Vania, Sardana Ivanova, Jennifer White, Rowan Hall Maudslay, Josef Valvoda, Ran Zmigrod, Paula Czarnowska, Irene Nikkarinen, Aelita Salchak, Brijesh Bhatt, Christopher Straughn, Zoey Liu, Jonathan North Washington, Yuval Pinter, Duygu Ataman, Marcin Wolinski, Totok Suhardijanto, Anna Yablonskaya, Niklas Stoehr, Hossep Dolatian, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Aryaman Arora, Richard J. Hatcher, Ritesh Kumar, Jeremiah Young, Daria Rodionova, Anastasia Yemelina, Taras Andrushko, Igor Marchenko, Polina Mashkovtseva, Alexandra Serova, Emily Prud’hommeaux, Maria Nepomniashchaya, Fausto Giunchiglia, Eleanor Chodroff, Mans Hulden, Miikka Silfverberg, Arya D. McCarthy, David Yarowsky, Ryan Cotterell, Reut Tsarfaty, Ekaterina Vylomova
| Challenge: | The Universal Morphology project provides broad-coverage instantiated morphological inflection tables for hundreds of diverse languages. |
| Approach: | They propose a language-independent feature schema for rich morphological annotation and a type-level resource of annotated data in diverse languages realizing that schema. |
| Outcome: | The proposed schema has added 66 new languages, including 24 endangered languages. |
Redcoat: A Collaborative Annotation Tool for Hierarchical Entity Typing (D19-3)
Copied to clipboard
| Challenge: | Recent advances in natural language processing (NLP) are fuelled by high quality annotated datasets. |
| Approach: | They introduce Redcoat, a web-based annotation tool that supports collaborative hierarchical entity typing. |
| Outcome: | The proposed annotation tool reduces the time it takes for project creators to set up and distribute projects to annotators and scales the workload depending on the number of active annotator. |
Discourse Relation-Enhanced Neural Coherence Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Existing work on coherence modeling has focused on integrating entity-based models. |
| Approach: | They propose a model that integrates text- and relation-based features for coherence assessment using position-aware attention and a visible matrix. |
| Outcome: | The proposed model improves baselines on two benchmarks and shows that relation features are important for coherence modeling. |
InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models have revolutionized the way summarization is generated. |
| Approach: | They propose a summarization model derived from GPT-3.5 through distillation that is compact and has comparable summarizing capabilities to GPT-3. |
| Outcome: | The proposed model outperforms the established best small models in prefix-tuning and full-data fine-tuned scenarios. |
We’re Afraid Language Models Aren’t Modeling Ambiguity (2023.emnlp-main)
Copied to clipboard
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, Yejin Choi
| Challenge: | Ambiguity is an intrinsic feature of natural language, allowing us to anticipate misunderstandings and revise our interpretations as listeners. |
| Approach: | They use AmbiEnt to capture ambiguity in a sentence and analyze it to evaluate pretrained LMs. |
| Outcome: | The proposed model can flag political claims in the wild that are misleading due to ambiguity. |
QuickGraph: A Rapid Annotation Tool for Knowledge Graph Extraction from Technical Text (2022.acl-demo)
Copied to clipboard
| Challenge: | Acquiring high-quality annotated corpora for complex multi-task information extraction (MT-IE) is an arduous and costly process for human-annotators. |
| Approach: | They propose a supervised MT-IE annotation tool built with indirect weak supervision and clustering to maximise annotator productivity. |
| Outcome: | The proposed tool is compared with existing tools in the field of MT-IE and aims to increase annotator productivity. |
LMGQS: A Large-scale Dataset for Query-focused Summarization (2023.findings-emnlp)
Copied to clipboard
Ruochen Xu, Song Wang, Yang Liu, Shuohang Wang, Yichong Xu, Dan Iter, Pengcheng He, Chenguang Zhu, Michael Zeng
| Challenge: | Lack of large-scale datasets for query-focused summarization hinders model development . lack of data limits the ability of QFS models to train robust neural models . |
| Approach: | They propose to generate a query for each summary sentence in a generic summarization annotation using a pretrained language model. |
| Outcome: | The proposed model achieves state-of-the-art zero-shot and supervised performance on multiple existing QFS benchmarks. |
Cloze-driven Pretraining of Self-attention Networks (D19-1)
Copied to clipboard
| Challenge: | Existing work on pretraining language models has used unidirectional (left-to-right) or bi-directional (both left-to right and right-to left) LMs with loss function. |
| Approach: | They propose a bi-directional transformer model that pretrains both directions of a large language-model-inspired self-attention cloze model and propose clozing to predict each word in the training data. |
| Outcome: | The proposed model performs well on GLUE and state of the art benchmarks consistent with BERT. |
Modeling Structural Similarities between Documents for Coherence Assessment with Graph Convolutional Networks (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods focus on a single document’s coherence patterns, ignoring the underlying correlation between documents. |
| Approach: | They propose a GCN-based coherence model that captures structural similarities between documents by mining subgraph patterns and a heterogeneous graph for the training corpus. |
| Outcome: | The proposed model outperforms baseline models on discourse coherence and automated essay scoring tasks. |
SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading (2024.emnlp-main)
Copied to clipboard
Tu Dinh, Carlos Mullov, Leonard Bärmann, Zhaolin Li, Danni Liu, Simon Reiß, Jueun Lee, Nathan Lerzer, Jianfeng Gao, Fabian Peller-Konrad, Tobias Röddiger, Alexander Waibel, Tamim Asfour, Michael Beigl, Rainer Stiefelhagen, Carsten Dachsbacher, Klemens Böhm, Jan Niehues
| Challenge: | Large Language Models (LLMs) are rapidly developing and are becoming more and more useful in scientific tasks. |
| Approach: | They propose to use LLM-as-a-judge to grade LLMs on SciEx to assess their ability on scientific tasks. |
| Outcome: | The proposed benchmarks show that the LLMs perform decently on free-form exams, achieving 0.948 Pearson correlation with expert grading. |
Unsupervised Multi-Granularity Summarization (2022.findings-emnlp)
Copied to clipboard
Ming Zhong, Yang Liu, Suyu Ge, Yuning Mao, Yizhu Jiao, Xingxing Zhang, Yichong Xu, Chenguang Zhu, Michael Zeng, Jiawei Han
| Challenge: | Experimental results confirm the substantial superiority of GranuSum on multi-granularity summarization over strong baselines. |
| Approach: | They propose to rank events by their salience and annotate a benchmark for GranuSum that contains multiple summaries at different granularities for each document cluster. |
| Outcome: | The proposed framework is capable of producing multi-granular summaries in unsupervised manner over strong baselines. |
Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models? (2023.findings-eacl)
Copied to clipboard
Jielin Qiu, William Han, Jiacheng Zhu, Mengdi Xu, Michael Rosenberg, Emerson Liu, Douglas Weber, Ding Zhao
| Challenge: | Recent advances in Large Language Models (LLMs) have shown powerful ability in various downstream applications. |
| Approach: | They propose an approach for cardiovascular disease diagnosis and automatic ECG diagnosis report generation. |
| Outcome: | The proposed approach generates high-quality cardiac diagnosis reports and achieves competitive zero-shot classification performance even compared with supervised baselines. |
Explanation-aware Soft Ensemble Empowers Large Language Model In-context Learning (2024.acl-long)
Copied to clipboard
Yue Yu, Jiaming Shen, Tianqi Liu, Zhen Qin, Jing Nathan Yan, Jialu Liu, Chao Zhang, Michael Bendersky
| Challenge: | Recent advances in natural language processing (NLP) have witnessed the remarkable capabilities of Large Language Models (LLMs). |
| Approach: | They propose an Explanation-Aware Soft Ensemble framework to empower in-context learning with Large language models. |
| Outcome: | The proposed framework can be used to enhance in-context learning on seven natural language understanding tasks and four varying-size LLMs. |
Task Compass: Scaling Multi-task Pre-training with Task Prefix (2022.findings-emnlp)
Copied to clipboard
Zhuosheng Zhang, Shuohang Wang, Yichong Xu, Yuwei Fang, Wenhao Yu, Yang Liu, Hai Zhao, Chenguang Zhu, Michael Zeng
| Challenge: | Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. |
| Approach: | They propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. |
| Outcome: | The proposed model can be used as a foundation backbone for a wide range of tasks and as augmentation tool for data augmentation with complementary tasks. |
Linguistic Cues for LLM-based Implicit Discourse Relation Classification (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been successful in many NLP tasks, but they struggle to capture subtle lexical relations between arguments. |
| Approach: | They propose a strategy that enriches arguments with explicit lexical-level semantic cues before fine-tuning. |
| Outcome: | The proposed approach improves F1 scores in cross-domain scenarios by more than 10 points compared to baselines. |
Joint Modeling of Entities and Discourse Relations for Coherence Assessment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on coherence modeling focuses on entity features or discourse relation features, with little attention given to combining the two. |
| Approach: | They propose two methods for jointly modeling entities and discourse relations for coherence assessment. |
| Outcome: | The proposed methods significantly improve the performance of coherence models on three benchmark datasets. |
Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data (2022.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that REtrieving from the traINing datA only can lead to significant gains on multiple NLG and NLU tasks. |
| Approach: | They propose to retrieve training instances from traINing datA and concatenate them with input to generate output. |
| Outcome: | The proposed method achieves state-of-the-art results on XSum, BigPatent, and CommonsenseQA. |
Proofread: Fixes All Errors with One Tap (2024.acl-demos)
Copied to clipboard
Renjie Liu, Yanxiang Zhang, Yun Zhu, Haicheng Sun, Yuanbo Zhang, Michael Huang, Shanqing Cai, Lei Meng, Shumin Zhai
| Challenge: | Extensive experiments on a human-labeled golden set showed our tuned PaLM2-XS model achieved 85.56% good ratio. |
| Approach: | They propose a two-stage tuning approach to acquire the dedicated Large Language Model for the feature, followed by a reinforcement learning approach for targeted refinement. |
| Outcome: | The proposed model achieves 85.56% good quality on Rewrite and proofread tasks on human-labeled golden sets. |