Papers by Bela Gipp
Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator (2025.coling-industry)
Copied to clipboard
| Challenge: | Established metrics such as ROUGE and BERTScore have a relatively low correlation with human judgments and fail to capture nuanced errors. |
| Approach: | They propose a framework that uses a three-step assessment of individual error types, multi-agent discussion for decision refinement, and feedback-based self-training to refine error definition understanding and alignment with human judgment. |
| Outcome: | The proposed framework achieves high correlation with human judgment and a consistent rating and adaptability to custom error guidelines. |
The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection (2025.findings-naacl)
Copied to clipboard
Tomáš Horych, Christoph Mandl, Terry Ruas, Andre Greiner-Petter, Bela Gipp, Akiko Aizawa, Timo Spinde
| Challenge: | Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. |
| Approach: | They propose to use Large Language Models to automate annotation process and train classifiers on large datasets. |
| Outcome: | The proposed model outperforms all of the annotator LLMs on two media bias benchmark datasets (BABE and BASIL) while maintaining data quality. |
BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages (2025.acl-long)
Copied to clipboard
Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nirmal Surange, Daniela Teodorescu, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino D. M. A. Ali, Ilseyar Alimova, Vladimir Araujo, Nikolay Babakov, Naomi Baes, Ana-Maria Bucur, Andiswa Bukula, Guanqun Cao, Rodrigo Tufiño, Rendi Chevi, Chiamaka Ijeoma Chukwuneke, Alexandra Ciobotaru, Daryna Dementieva, Murja Sani Gadanya, Robert Geislinger, Bela Gipp, Oumaima Hourrane, Oana Ignat, Falalu Ibrahim Lawan, Rooweither Mabuya, Rahmad Mahendra, Vukosi Marivate, Alexander Panchenko, Andrew Piper, Charles Henrique Porto Ferreira, Vitaly Protasov, Samuel Rutunda, Manish Shrivastava, Aura Cristina Udrea, Lilian Diana Awuor Wanzare, Sophie Wu, Florian Valentin Wunderlich, Hanif Muhammad Zhafran, Tianhui Zhang, Yi Zhou, Saif M. Mohammad
| Challenge: | Emotion recognition is an umbrella term for several NLP tasks, but most work on high-resource languages has focused on low-resourced languages. |
| Approach: | They propose to use emotion recognition to describe perceived emotions in 28 different languages and across several domains to identify and annotate the datasets. |
| Outcome: | The proposed datasets cover low-resource languages from Africa, Asia, Eastern Europe, and Latin America, with instances labeled by fluent speakers. |
What’s under the hood: Investigating Automatic Metrics on Meeting Summarization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics do not capture meeting-specific errors, leading to ineffective assessment. |
| Approach: | They examine the relationship between established metrics and human evaluations to determine what challenges and errors are captured by correlating metric scores with human evaluation. |
| Outcome: | The proposed measures show weak correlations with human evaluations and a third of the correlations show error masking. |
Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on the detection and aggregation of media bias lack a gold standard data set and high context dependencies. |
| Approach: | They propose to use a data set to identify media bias by word and sentence level . they propose to train a model to detect bias-inducing sentences in news articles automatically . |
| Outcome: | The proposed model outperforms existing methods on a large corpus of labels on the word and sentence level. |
Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies that analyze unseen domains vary translation systems, annotators, or evaluation conditions, confounding domain effects with human annotation noise. |
| Approach: | They propose to use human error span annotations to evaluate translations of six translation systems across one seen news domain and two unseen technical domains to address these biases. |
| Outcome: | The proposed model improves on the human annotations in two unseen domains and on the news domains. |
Voting or Consensus? Decision-Making in Multi-Agent Debate (2025.findings-acl)
Copied to clipboard
| Challenge: | Increasing the number of agents improves performance, while more discussion rounds before voting reduces it. |
| Approach: | They propose two new methods to improve multi-agent debates by increasing agent diversity and reducing discussion rounds before voting. |
| Outcome: | The proposed methods improve task performance by up to 3.3% with AAD and up to 7.4% with CI. |
Aspect-based Document Similarity for Research Papers (2020.coling-main)
Copied to clipboard
| Challenge: | Traditional document similarity measures do not consider in what aspects two documents are similar. |
| Approach: | They extend document similarity with aspect information by performing a pairwise document classification task. |
| Outcome: | The proposed approach is best performing on 172,073 research paper pairs from the ACL Anthology and CORD-19 corpus. |
ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents (2026.acl-long)
Copied to clipboard
| Challenge: | Visually rich documents (VRDs) combine text, tables, and figures within complex, semantically structured layouts. |
| Approach: | They propose a multi-turn reinforcement learning framework that fine-tunes VLMs as interactive agents capable of actively navigating long, visually rich documents. |
| Outcome: | The proposed framework achieves state-of-the-art on five long-document benchmarks. |
MALLM: Multi-Agent Large Language Models Framework (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Multi-agent debate (MAD) has demonstrated the ability to augment collective intelligence by scaling test-time compute and leveraging expertise. |
| Approach: | They propose an open-source framework that enables systematic analysis of multi-agent debates. |
| Outcome: | The proposed framework enables systematic analysis of multi-agent debate components. |
Neural Machine Translation for Mathematical Formulae (2023.acl-long)
Copied to clipboard
| Challenge: | a recent paper examines the problem of neural machine translation of mathematical formulae between ambiguous presentation languages and unambiguous content languages. |
| Approach: | They perform translation tasks from LaTeX to Mathematica and from La TeX into semantic LaTaX using convolutional sequence-to-sequence networks. |
| Outcome: | The proposed translations achieve 95.1% and 90.7% exact matches between the two languages. |
D3: A Massive Dataset of Scholarly Metadata for Analyzing the State of Computer Science Research (2022.lrec-1)
Copied to clipboard
| Challenge: | DBLP is the largest open-access repository of scientific articles on computer science and provides metadata associated with publications, authors, and venues. |
| Approach: | They extracted metadata from more than 6 million DBLP publications to create the DB3 Discovery Dataset (D3) . they found that computer science is a growing research field (15% annually), with an active and collaborative researcher community. |
| Outcome: | The DBLP Discovery Dataset (D3) can be used to identify trends in research activity, productivity, focus, bias, accessibility, and impact of computer science research. |
Citation Amnesia: On The Recency Bias of NLP and Other Academic Fields (2025.coling-main)
Copied to clipboard
| Challenge: | citation age is a key factor in determining whether older works are cited in scientific journals or not. |
| Approach: | They examine the tendency of NLP to cite older work across 20 fields of study over 43 years (1980–2023) . they put NLP’s propensity to citation older work in the context of these 20 other fields to see whether differences can be observed . |
| Outcome: | The trend is strongest in NLP and ML research (-12.8% and -5.5% in citation age from previous peaks) |
What’s Wrong? Refining Meeting Summaries with LLM Feedback (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for meeting summarization are limited and lack the robustness and context-based accuracy needed to maintain relevance. |
| Approach: | They propose a multi-LLM correction approach for meeting summarization using a two-phase process that mimics the human review process: mistake identification and summary refinement. |
| Outcome: | The proposed approach improves the quality of a given meeting summarization measured by relevance, informativeness, conciseness, and coherence. |
Paraphrase Types Elicit Prompt Engineering Capabilities (2024.emnlp-main)
Copied to clipboard
| Challenge: | Until now, it has been unknown how variations in the linguistic expression of prompts affect language models. |
| Approach: | They evaluate which linguistic features influence models through paraphrase types . they found that changes in morphology and lexicon showed promise in improving prompts . |
| Outcome: | The results show that paraphrases can improve language models' performance . the authors show that changes in morphology and lexicon can improve prompts . |
You need to MIMIC to get FAME: Solving Meeting Transcript Scarcity with Multi-Agent Conversations (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing tools for meeting summarization are limited due to privacy and expensive manual annotation. |
| Approach: | They propose a meeting synthesis framework that generates meeting transcripts on a given knowledge source by defining psychologically grounded participant profiles, outlining the conversation, and orchestrating a large language model (LLM) debate. |
| Outcome: | The proposed framework generates meeting transcripts on a given knowledge source by defining psychologically grounded participant profiles, outlining the conversation, and orchestrating a large language model debate. |
TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has focused on the (un)intended leakage of sensitive information through LLM outputs. |
| Approach: | They propose a threat model that embeds context information into natural-looking outputs via linguistic steganography without requiring explicit control over inference inputs. |
| Outcome: | The proposed model transmits 32-bit secrets with 87% accuracy on held-out prompts and can reach over 97% accuracy using majority voting across three generations. |
How Large Language Models are Transforming Machine-Paraphrase Plagiarism (2022.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive paraphrasing tools can be used to generate convincing plagiarized texts with minimal effort. |
| Approach: | They evaluate the detection performance of large autoregressive models for machine-paraphrase generation on scientific articles from arXiv, student theses, and Wikipedia. |
| Outcome: | The proposed models generate paraphrases indistinguishable from original work and human experts rate the quality of generated examples as high as originals. |
Re-FRAME the Meeting Summarization SCOPE: Fact-Based Summarization and Personalization via Questions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | FRAME reframes summarization as a semantic enrichment task . SCOPE is a reason-out-loud protocol that has the model build a reasoning trace . |
| Approach: | They propose a modular pipeline that reframes summarization as a semantic enrichment task. |
| Outcome: | The proposed pipeline reduces hallucinations and omissions by 2 out of 5 points . SCOPE improves knowledge fit and goal alignment over prompt-only baselines . |
We are Who We Cite: Bridges of Influence Between Natural Language Processing and Other Academic Fields (2023.emnlp-main)
Copied to clipboard
| Challenge: | In this paper, we quantify the degree of influence between 23 fields of study and NLP (on each other) |
| Approach: | They quantify the degree of influence between 23 fields of study and NLP on each other . they find that cross-field engagement of NLP has declined from 0.58 in 1980 to 0.31 in 2022 . |
| Outcome: | The proposed Citation Field Diversity Index (CFDI) has declined from 0.58 in 1980 to 0.31 in 2022, the authors show . |
Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prior work relies on discrete citation relations to generate contrast samples, but discrete ones enforce a hard cut-off to similarity. |
| Approach: | They propose to use nearest neighbor sampling to learn continuous similarity and to sample hard-to-learn negatives and positives by controlling the sampling margin between them. |
| Outcome: | The proposed method outperforms the state-of-the-art on the SciDocs benchmark and can train (or tune) language models sample-efficiently. |
Tell me what I need to know: Exploring LLM-based (Personalized) Abstractive Multi-Source Meeting Summarization (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for meeting summarization rely on transcripts and generate generic summaries, failing to contextualize long discussions and to tailor information to individual preferences and productivity requirements. |
| Approach: | They propose a multi-source approach that considers supplementary materials and generates a summary from this enriched transcript. |
| Outcome: | The proposed model increases summary relevance by 9% and produces more content-rich outputs. |
Towards Evaluation of Cross-document Coreference Resolution Models Using Datasets with Diverse Annotation Schemes (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing cross-document coreference resolution (CDCR) datasets contain event-centric coreference chains of events and entities with identity relations. |
| Approach: | They propose to use a phrasing diversity metric to evaluate lexical diversity of CDCR datasets . they propose to combine CDCR annotation schemes with multiple properties of the coreference chains . |
| Outcome: | The proposed phrasing diversity metric evaluates the CDCR datasets with higher precision. |
Towards Human Understanding of Paraphrase Types in Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Current paraphrase evaluations of language models use binary approaches, offering limited interpretability of specific text changes. |
| Approach: | They introduce a dataset of 800 sentence-level and word-level annotations by 15 annotators and a human preference ranking of paraphrases with different types. |
| Outcome: | The proposed model can generate simple APTs, but struggle with complex structures (e.g., subordination changes). |
Stay Focused: Problem Drift in Multi-Agent Debate (2026.findings-eacl)
Copied to clipboard
| Challenge: | Multi-agent debates have shown promise for solving knowledge and reasoning tasks, but they are limited when solving complex problems that require longer reasoning chains. |
| Approach: | They propose a method to detect problem drift and propose 'driFTJudge' which mitigates 31% of problem drift cases. |
| Outcome: | The proposed method mitigates 31% of problem drift cases and is based on a set of ten tasks across ten different tasks. |
SPaRC: A Spatial Pathfinding Reasoning Challenge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing reasoning datasets saturate and fail to test abstract, multi-step problems, especially pathfinding and complex rule constraint satisfaction. |
| Approach: | They propose to use a spatial few-shot grid to evaluate spatial and rule-based reasoning with 1,000 2D grid puzzles. |
| Outcome: | The proposed model can be used to evaluate spatial reasoning and improve its accuracy. |
Text-Guided Image Clustering (2024.eacl-long)
Copied to clipboard
Andreas Stephan, Lukas Miklautz, Kevin Sidak, Jan Philip Wahle, Bela Gipp, Claudia Plant, Benjamin Roth
| Challenge: | Current image clustering methods neglect the use of generated textual descriptions. |
| Approach: | They propose to use image captioning and visual question-answering to cluster images . they propose a new approach to inject task- or domain knowledge into image clustering . |
| Outcome: | The proposed method outperforms existing methods on eight image clustering datasets. |
Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling (2026.acl-srw)
Copied to clipboard
| Challenge: | Inference methods that prioritize raw performance over cost-effective compute usage are not efficient for real-world applications. |
| Approach: | They evaluate inference scaling strategies to determine their computational efficiency tradeoffs . they find debate and mixture-of-agents outperform self-consistency by 1.3% and 2.7% points . |
| Outcome: | The proposed scaling strategies outperform self-consistency, self-refinement, multi-agent debate and mixture-of-a agents on reasoning tasks. |
Paraphrase Types for Generation and Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to paraphrase generation and detection ignore the intricate linguistic properties of language. |
| Approach: | They propose two tasks to consider specific linguistic perturbations at particular text positions. |
| Outcome: | The proposed tasks address the shortcoming of ignoring the linguistic properties of language. |
Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industry (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Recent trends in NLP utilize knowledge graphs to enhance pretrained language models by incorporating additional knowledge from the graph structures to learn domain-specific terminology or relationships between documents that might otherwise be overlooked. |
| Approach: | They propose to use graph-aware neighborhood contrastive learning methodology SciNCL to enhance pretrained language models by incorporating additional knowledge from graph structures. |
| Outcome: | The proposed graph-aware neighborhood contrastive learning methodology outperforms a state-of-the-art mE5-large text encoder on the process industry text embedding benchmark while having 3 times fewer parameters. |
MAGPIE: Multi-Task Analysis of Media-Bias Generalization with Pre-Trained Identification of Expressions (2024.lrec-main)
Copied to clipboard
Tomáš Horych, Martin Paul Wessel, Jan Philip Wahle, Terry Ruas, Jerome Waßmuth, André Greiner-Petter, Akiko Aizawa, Bela Gipp, Timo Spinde
| Challenge: | Existing approaches to media bias detection lack generalizability, resulting in limited generalizarability. |
| Approach: | They propose a large-scale multi-task pre-training approach specifically tailored for media bias detection that can be used to train 59 bias-related tasks. |
| Outcome: | The proposed approach outperforms existing methods on the BABE dataset with a relative improvement of 3.3% F1-score. |