Papers by Benjamin Liu
Core: Robust Factual Precision with Informative Sub-Claim Identification (2025.findings-acl)
Copied to clipboard
Zhengping Jiang, Jingyu Zhang, Nathaniel Weir, Seth Ebner, Miriam Wanner, Kate Sanders, Daniel Khashabi, Anqi Liu, Benjamin Van Durme
| Challenge: | Using the Decompose-Then-Verify framework, such as FActScore, can be manipulated by adding obvious or repetitive subclaims to artificially inflate scores. |
| Approach: | They propose a decomposition-based tool called Core to filter subclaims based on their uniqueness and informativeness. |
| Outcome: | The proposed evaluation framework supports easy and modular use of Core and various decomposition strategies. |
Joint Goal Segmentation and Goal Success Prediction on Multi-Domain Conversations (2022.coling-1)
Copied to clipboard
| Challenge: | Existing metrics to measure the performance of conversational AI assistants are difficult to establish due to their slow nature. |
| Approach: | They propose an automatic dialogue evaluation framework that performs goal segmentation and success prediction by adding multi-task learning heads. |
| Outcome: | The proposed model achieves on-par with human annotation compared to a gold annotation benchmark. |
KEPLET: Knowledge-Enhanced Pretrained Language Model with Topic Entity Awareness (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have shown their superiority by pre-training on unstructured text corpus and then fine-tuning on downstream tasks. |
| Approach: | They propose a Knowledge-Enhanced Pre-trained LanguagE model with Topic entity awareness that incorporates the interactions between tokens and mentioned entities in pre-training. |
| Outcome: | The proposed model incorporates the interactions between tokens and mentioned entities in pre-training and is more effective on entity-centric tasks. |
Calibrating Zero-shot Cross-lingual (Un-)structured Predictions (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing need for model calibration when natural language models are deployed in critical tasks. |
| Approach: | They compare model calibration methods in a context of zero-shot cross-lingual transfer with pre-trained language models. |
| Outcome: | The proposed method fails to calibrate more complex confidence estimations in structured predictions compared to expressive alternatives like Gaussian Process Calibration. |
RORA: Robust Free-Text Rationale Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing metrics rely on degree to which rationale supports a label, but they fail to evaluate rationales that inadvertently leak the label. |
| Approach: | They propose a RObust free-text RAtionale evaluation against label leakage that quantifies the new information supplied by a rationale to justify the label. |
| Outcome: | The proposed evaluation outperforms existing methods in evaluating human-written, synthetic, or model-generated rationales, particularly demonstrating robustness against label leakage. |
Jargon: A Suite of Language Models and Evaluation Tasks for French Specialized Domains (2024.lrec-main)
Copied to clipboard
Vincent Segonne, Aidan Mannion, Laura Cristina Alonzo Canul, Alexandre Daniel Audibert, Xingyu Liu, Cécile Macaire, Adrien Pupier, Yongxin Zhou, Mathilde Aguiar, Felix E. Herron, Magali Norré, Massih R Amini, Pierrette Bouillon, Iris Eshkol-Taravella, Emmanuelle Esperança-Rodier, Thomas François, Lorraine Goeuriot, Jérôme Goulian, Mathieu Lafourcade, Benjamin Lecouteux, François Portet, Fabien Ringeval, Vincent Vandeghinste, Maximin Coavoux, Marco Dinarelli, Didier Schwab
| Challenge: | Pretrained language models are the de facto backbone of most state-of-the-art NLP systems. |
| Approach: | They propose a family of domain-specific pretrained PLMs for French focusing on three important domains: transcribed speech, medicine, and law. |
| Outcome: | The proposed models perform better on transcribed speech, medicine, and law domains than state-of-the-art models on a diverse set of tasks and datasets. |
Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting (2023.findings-emnlp)
Copied to clipboard
Benjamin Yan, Ruochen Liu, David Kuo, Subathra Adithan, Eduardo Reis, Stephen Kwak, Vasantha Venugopal, Chloe O’Connell, Agustina Saenz, Pranav Rajpurkar, Michael Moor
| Challenge: | Existing methods for generating reports from medical images conflate the content of the report with its style, which can lead to inaccurate reports. |
| Approach: | They propose a two-step approach to generate radiology reports from medical images using large language models and a graph representation of reports. |
| Outcome: | The proposed approach improves the performance of human evaluations with clinical raters. |
D2GCLF: Document-to-Graph Classifier for Legal Document Classification (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods learn latent representations for each document by considering the semantics and themes of the documents. |
| Approach: | They propose a document-to-graph classifier which extracts facts as relations between key participants in a law case and represents a legal document with four relation graphs. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on a real-world legal document dataset. |
RATIONALYST: Pre-training Process-Supervision for Improving Reasoning (2025.acl-long)
Copied to clipboard
Dongwei Jiang, Guoxuan Wang, Yining Lu, Andrew Wang, Jingyu Zhang, Chuyu Liu, Benjamin Van Durme, Daniel Khashabi
| Challenge: | RATIONALYST is a model for process-supervision of reasoning based on pretraining on rationale annotations extracted from unlabeled data. |
| Approach: | They propose a model for process-supervision of reasoning based on pre-training on rationale annotations extracted from unlabeled data. |
| Outcome: | RATIONALYST improves reasoning accuracy by 3.9% on representative reasoning benchmarks. |
Closing the Spatial Execution Gap in Digital Whiteboards via Verifiable Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models suffer from a fundamental Spatial Execution Gap, failing to translate visual semantics into precise, schema-valid coordinate operations in interactive environments. |
| Approach: | They propose a pipeline that leverages Group Relative Policy Optimization to enforce a strict Identify-Reason-Verify protocol and train on execution-verifiable rewards. |
| Outcome: | The proposed pipeline outperforms a state-of-the-art frontier model by 16.75% in operation accuracy. |
SKGSum: Structured Knowledge-Guided Document Summarization (2024.findings-acl)
Copied to clipboard
Qiqi Wang, Ruofan Wang, Kaiqi Zhao, Robert Amor, Benjamin Liu, Jiamou Liu, Xianda Zheng, Zijian Huang
| Challenge: | Existing summarization methods ignore the importance of summary structure, resulting in summaries that emphasize the most prominent information while omitting essential details from other sections. |
| Approach: | They propose a method that uses automatically extracted summary points to generate summaries. |
| Outcome: | The proposed methods improve quality and BERTScore of summaries and broaden the types of documents that can be effectively summarized. |
M3T: A New Benchmark Dataset for Multi-Modal Document-Level Machine Translation (2024.naacl-short)
Copied to clipboard
Benjamin Hsu, Xiaoyu Liu, Huayang Li, Yoshinari Fujinuma, Maria Nadejde, Xing Niu, Ron Litman, Yair Kittenplon, Raghavendra Pappagari
| Challenge: | Document translation is a challenge for machine translation systems that focus on textual content at the sentence level, ignoring global context and visual layout structure. |
| Approach: | They propose a benchmark dataset to evaluate document-level NMT systems . they use visual cues to preserve reading order and contiguous blocks of text . |
| Outcome: | The proposed benchmarks assess document-level NMT systems on the comprehensive task of translating semi-structured documents. |