Papers by Terry Ruas
The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection (2025.findings-naacl)
Copied to clipboard
Tomáš Horych, Christoph Mandl, Terry Ruas, Andre Greiner-Petter, Bela Gipp, Akiko Aizawa, Timo Spinde
| Challenge: | Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. |
| Approach: | They propose to use Large Language Models to automate annotation process and train classifiers on large datasets. |
| Outcome: | The proposed model outperforms all of the annotator LLMs on two media bias benchmark datasets (BABE and BASIL) while maintaining data quality. |
BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages (2025.acl-long)
Copied to clipboard
Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nirmal Surange, Daniela Teodorescu, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino D. M. A. Ali, Ilseyar Alimova, Vladimir Araujo, Nikolay Babakov, Naomi Baes, Ana-Maria Bucur, Andiswa Bukula, Guanqun Cao, Rodrigo Tufiño, Rendi Chevi, Chiamaka Ijeoma Chukwuneke, Alexandra Ciobotaru, Daryna Dementieva, Murja Sani Gadanya, Robert Geislinger, Bela Gipp, Oumaima Hourrane, Oana Ignat, Falalu Ibrahim Lawan, Rooweither Mabuya, Rahmad Mahendra, Vukosi Marivate, Alexander Panchenko, Andrew Piper, Charles Henrique Porto Ferreira, Vitaly Protasov, Samuel Rutunda, Manish Shrivastava, Aura Cristina Udrea, Lilian Diana Awuor Wanzare, Sophie Wu, Florian Valentin Wunderlich, Hanif Muhammad Zhafran, Tianhui Zhang, Yi Zhou, Saif M. Mohammad
| Challenge: | Emotion recognition is an umbrella term for several NLP tasks, but most work on high-resource languages has focused on low-resourced languages. |
| Approach: | They propose to use emotion recognition to describe perceived emotions in 28 different languages and across several domains to identify and annotate the datasets. |
| Outcome: | The proposed datasets cover low-resource languages from Africa, Asia, Eastern Europe, and Latin America, with instances labeled by fluent speakers. |
What’s under the hood: Investigating Automatic Metrics on Meeting Summarization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics do not capture meeting-specific errors, leading to ineffective assessment. |
| Approach: | They examine the relationship between established metrics and human evaluations to determine what challenges and errors are captured by correlating metric scores with human evaluation. |
| Outcome: | The proposed measures show weak correlations with human evaluations and a third of the correlations show error masking. |
Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on the detection and aggregation of media bias lack a gold standard data set and high context dependencies. |
| Approach: | They propose to use a data set to identify media bias by word and sentence level . they propose to train a model to detect bias-inducing sentences in news articles automatically . |
| Outcome: | The proposed model outperforms existing methods on a large corpus of labels on the word and sentence level. |
Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies that analyze unseen domains vary translation systems, annotators, or evaluation conditions, confounding domain effects with human annotation noise. |
| Approach: | They propose to use human error span annotations to evaluate translations of six translation systems across one seen news domain and two unseen technical domains to address these biases. |
| Outcome: | The proposed model improves on the human annotations in two unseen domains and on the news domains. |
Voting or Consensus? Decision-Making in Multi-Agent Debate (2025.findings-acl)
Copied to clipboard
| Challenge: | Increasing the number of agents improves performance, while more discussion rounds before voting reduces it. |
| Approach: | They propose two new methods to improve multi-agent debates by increasing agent diversity and reducing discussion rounds before voting. |
| Outcome: | The proposed methods improve task performance by up to 3.3% with AAD and up to 7.4% with CI. |
Aspect-based Document Similarity for Research Papers (2020.coling-main)
Copied to clipboard
| Challenge: | Traditional document similarity measures do not consider in what aspects two documents are similar. |
| Approach: | They extend document similarity with aspect information by performing a pairwise document classification task. |
| Outcome: | The proposed approach is best performing on 172,073 research paper pairs from the ACL Anthology and CORD-19 corpus. |
ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents (2026.acl-long)
Copied to clipboard
| Challenge: | Visually rich documents (VRDs) combine text, tables, and figures within complex, semantically structured layouts. |
| Approach: | They propose a multi-turn reinforcement learning framework that fine-tunes VLMs as interactive agents capable of actively navigating long, visually rich documents. |
| Outcome: | The proposed framework achieves state-of-the-art on five long-document benchmarks. |
MALLM: Multi-Agent Large Language Models Framework (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Multi-agent debate (MAD) has demonstrated the ability to augment collective intelligence by scaling test-time compute and leveraging expertise. |
| Approach: | They propose an open-source framework that enables systematic analysis of multi-agent debates. |
| Outcome: | The proposed framework enables systematic analysis of multi-agent debate components. |
D3: A Massive Dataset of Scholarly Metadata for Analyzing the State of Computer Science Research (2022.lrec-1)
Copied to clipboard
| Challenge: | DBLP is the largest open-access repository of scientific articles on computer science and provides metadata associated with publications, authors, and venues. |
| Approach: | They extracted metadata from more than 6 million DBLP publications to create the DB3 Discovery Dataset (D3) . they found that computer science is a growing research field (15% annually), with an active and collaborative researcher community. |
| Outcome: | The DBLP Discovery Dataset (D3) can be used to identify trends in research activity, productivity, focus, bias, accessibility, and impact of computer science research. |
Paraphrase Types Elicit Prompt Engineering Capabilities (2024.emnlp-main)
Copied to clipboard
| Challenge: | Until now, it has been unknown how variations in the linguistic expression of prompts affect language models. |
| Approach: | They evaluate which linguistic features influence models through paraphrase types . they found that changes in morphology and lexicon showed promise in improving prompts . |
| Outcome: | The results show that paraphrases can improve language models' performance . the authors show that changes in morphology and lexicon can improve prompts . |
DimABSA: Building Multilingual and Multidomain Datasets for Dimensional Aspect-Based Sentiment Analysis (2026.acl-long)
Copied to clipboard
Lung-Hao Lee, Liang-Chih Yu, Natalia V Loukachevitch, Ilseyar Alimova, Alexander Panchenko, Tzu-Mi Lin, Zhe-Yu Xu, Jian-Yu Zhou, Guangmin Zheng, Jin Wang, Sharanya Awasthi, Jonas Becker, Jan Philip Wahle, Terry Ruas, Shamsuddeen Hassan Muhammad, Saif M. Mohammad
| Challenge: | Existing ABSA research relies on coarse-grained categorical labels, which limits its ability to capture nuanced affective states. |
| Approach: | They propose a dimensional approach that represents sentiment with continuous valence–arousal (VA) scores, enabling fine-grained analysis at both the aspect and sentiment levels. |
| Outcome: | The proposed approach represents sentiment with continuous valence–arousal (VA) scores, enabling fine-grained analysis at both the aspect and sentiment levels. |
You need to MIMIC to get FAME: Solving Meeting Transcript Scarcity with Multi-Agent Conversations (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing tools for meeting summarization are limited due to privacy and expensive manual annotation. |
| Approach: | They propose a meeting synthesis framework that generates meeting transcripts on a given knowledge source by defining psychologically grounded participant profiles, outlining the conversation, and orchestrating a large language model (LLM) debate. |
| Outcome: | The proposed framework generates meeting transcripts on a given knowledge source by defining psychologically grounded participant profiles, outlining the conversation, and orchestrating a large language model debate. |
TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has focused on the (un)intended leakage of sensitive information through LLM outputs. |
| Approach: | They propose a threat model that embeds context information into natural-looking outputs via linguistic steganography without requiring explicit control over inference inputs. |
| Outcome: | The proposed model transmits 32-bit secrets with 87% accuracy on held-out prompts and can reach over 97% accuracy using majority voting across three generations. |
How Large Language Models are Transforming Machine-Paraphrase Plagiarism (2022.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive paraphrasing tools can be used to generate convincing plagiarized texts with minimal effort. |
| Approach: | They evaluate the detection performance of large autoregressive models for machine-paraphrase generation on scientific articles from arXiv, student theses, and Wikipedia. |
| Outcome: | The proposed models generate paraphrases indistinguishable from original work and human experts rate the quality of generated examples as high as originals. |
Re-FRAME the Meeting Summarization SCOPE: Fact-Based Summarization and Personalization via Questions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | FRAME reframes summarization as a semantic enrichment task . SCOPE is a reason-out-loud protocol that has the model build a reasoning trace . |
| Approach: | They propose a modular pipeline that reframes summarization as a semantic enrichment task. |
| Outcome: | The proposed pipeline reduces hallucinations and omissions by 2 out of 5 points . SCOPE improves knowledge fit and goal alignment over prompt-only baselines . |
We are Who We Cite: Bridges of Influence Between Natural Language Processing and Other Academic Fields (2023.emnlp-main)
Copied to clipboard
| Challenge: | In this paper, we quantify the degree of influence between 23 fields of study and NLP (on each other) |
| Approach: | They quantify the degree of influence between 23 fields of study and NLP on each other . they find that cross-field engagement of NLP has declined from 0.58 in 1980 to 0.31 in 2022 . |
| Outcome: | The proposed Citation Field Diversity Index (CFDI) has declined from 0.58 in 1980 to 0.31 in 2022, the authors show . |
Tell me what I need to know: Exploring LLM-based (Personalized) Abstractive Multi-Source Meeting Summarization (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for meeting summarization rely on transcripts and generate generic summaries, failing to contextualize long discussions and to tailor information to individual preferences and productivity requirements. |
| Approach: | They propose a multi-source approach that considers supplementary materials and generates a summary from this enriched transcript. |
| Outcome: | The proposed model increases summary relevance by 9% and produces more content-rich outputs. |
Stay Focused: Problem Drift in Multi-Agent Debate (2026.findings-eacl)
Copied to clipboard
| Challenge: | Multi-agent debates have shown promise for solving knowledge and reasoning tasks, but they are limited when solving complex problems that require longer reasoning chains. |
| Approach: | They propose a method to detect problem drift and propose 'driFTJudge' which mitigates 31% of problem drift cases. |
| Outcome: | The proposed method mitigates 31% of problem drift cases and is based on a set of ten tasks across ten different tasks. |
SPaRC: A Spatial Pathfinding Reasoning Challenge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing reasoning datasets saturate and fail to test abstract, multi-step problems, especially pathfinding and complex rule constraint satisfaction. |
| Approach: | They propose to use a spatial few-shot grid to evaluate spatial and rule-based reasoning with 1,000 2D grid puzzles. |
| Outcome: | The proposed model can be used to evaluate spatial reasoning and improve its accuracy. |
Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling (2026.acl-srw)
Copied to clipboard
| Challenge: | Inference methods that prioritize raw performance over cost-effective compute usage are not efficient for real-world applications. |
| Approach: | They evaluate inference scaling strategies to determine their computational efficiency tradeoffs . they find debate and mixture-of-agents outperform self-consistency by 1.3% and 2.7% points . |
| Outcome: | The proposed scaling strategies outperform self-consistency, self-refinement, multi-agent debate and mixture-of-a agents on reasoning tasks. |
Paraphrase Types for Generation and Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to paraphrase generation and detection ignore the intricate linguistic properties of language. |
| Approach: | They propose two tasks to consider specific linguistic perturbations at particular text positions. |
| Outcome: | The proposed tasks address the shortcoming of ignoring the linguistic properties of language. |
The Elephant in the Room: Analyzing the Presence of Big Tech in Natural Language Processing Research (2023.acl-long)
Copied to clipboard
Mohamed Abdalla, Jan Philip Wahle, Terry Ruas, Aurélie Névéol, Fanny Ducel, Saif Mohammad, Karen Fort
| Challenge: | Recent advances in deep learning methods for natural language processing (NLP) have created new business opportunities and made NLP research critical for industry development. |
| Approach: | They examine industry presence in the field since the early 90s and characterize it using a corpus of 78,187 NLP publications and 701 resumes of NLP publication authors. |
| Outcome: | The authors find that industry presence among NLP authors has been steady before a steep increase over the past five years (180% growth from 2017 to 2022). |
MAGPIE: Multi-Task Analysis of Media-Bias Generalization with Pre-Trained Identification of Expressions (2024.lrec-main)
Copied to clipboard
Tomáš Horych, Martin Paul Wessel, Jan Philip Wahle, Terry Ruas, Jerome Waßmuth, André Greiner-Petter, Akiko Aizawa, Bela Gipp, Timo Spinde
| Challenge: | Existing approaches to media bias detection lack generalizability, resulting in limited generalizarability. |
| Approach: | They propose a large-scale multi-task pre-training approach specifically tailored for media bias detection that can be used to train 59 bias-related tasks. |
| Outcome: | The proposed approach outperforms existing methods on the BABE dataset with a relative improvement of 3.3% F1-score. |