Papers by Steffen Eger

Vec2Sent: Probing Sentence Embeddings with Natural Language Generation (2020.coling-main)

Challenge:	DiscoScore is a parametrized discourse metric that uses BERT to model discourse coherence . it is weak when operated at system level, and is therefore not reliable in a way to spot improvements .
Approach:	They propose a parametrized discourse metric which uses BERT to model discourse coherence from different perspectives.
Outcome:	The proposed model outperforms existing models on document-level machine translation and summarization.

Copied to clipboard

Martin Kerscher, Steffen Eger

Representation Learning

Challenge:	a new unsupervised probing task is able to retrieve black-box sentence embeddings . a variety of problems surround probing tasks, including manual construing .
Approach:	They propose a method to generate black-box sentence embeddings by conditionally generating from them . they also illustrate how the language generated from different encoders differs .
Outcome:	The proposed probing task improves the performance of black-box sentence embeddings . the proposed task is based on a conditional natural language generation approach .

Better than Average: Paired Evaluation of NLP systems (2021.acl-long)

Copied to clipboard

Maxime Peyrard, Wei Zhao, Steffen Eger, Robert West

Challenge:	Evaluation in NLP is usually done by comparing the scores of competing systems . averaging scores independently and declaring the best system is difficult .
Approach:	They examine the use of averages to aggregate evaluation scores into a final number . they argue that the average ignores the pairing arising from the fact that systems are evaluated on the same test instances.
Outcome:	The proposed method ignores the pairing arising from the fact that systems are evaluated on the same test instances.

Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks (D18-1)

Copied to clipboard

Steffen Eger, Paul Youssef, Iryna Gurevych

Challenge:	Activation functions are nonlinearities which have been attributed to the success story of deep learning.
Approach:	They propose to use a penalized tanh function to replace the sigmoid and tansh gates in LSTM cells and to improve the performance of the activation function.
Outcome:	The proposed activation function performs best on all tasks and can replace the sigmoid and tanh gates in LSTM cells.

Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications (P19-1)

Copied to clipboard

Wei Zhao, Haiyun Peng, Steffen Eger, Erik Cambria, Min Yang

Challenge:	Existing approaches to generalize deep neural networks are datahungry and generalize poorly from small datasets.
Approach:	They propose an agreement score to evaluate routing processes at instance-level and an adaptive optimizer to enhance routing.
Outcome:	The proposed approach improves on two NLP tasks and in low-resource settings with few training instances.

Fine-Grained Detection of Solidarity for Women and Migrants in 155 Years of German Parliamentary Debates (2024.emnlp-main)

Copied to clipboard

Aida Kostikova, Dominik Beese, Benjamin Paassen, Ole Pütz, Gregor Wiedemann, Steffen Eger

LiTransProQA: An LLM-based Literary Translation Evaluation Metric with Professional Question Answering (2025.emnlp-main)

Challenge:	Traditionally, solidarity relied on common identity and reciprocity, potentially excluding out-groups like migrants.
Approach:	They examine the frequency of (anti-)solidarity towards women and migrants in German parliamentary debates between 1867 and 2022.
Outcome:	The proposed model outperforms other models in the analysis of 2,864 text snippets and finds that solidarity with migrants outweighs anti-solidarity but frequencies and solidarity types shift over time.

Copied to clipboard

Ran Zhang, Wei Zhao, Lieve Macken, Steffen Eger

Machine Translation

Challenge:	Existing evaluation metrics for literature prioritize mechanical accuracy over artistic expression . this bias could result in an irreversible decline in translation quality and cultural authenticity .
Approach:	They propose a novel, reference-free, LLM-based question-answering framework for literary translation evaluation.
Outcome:	a novel, reference-free, LLM-based question-answering framework is developed for literary translation evaluation.

PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics (2025.naacl-long)

Copied to clipboard

Daniil Larionov, Steffen Eger

ArgumenText: Searching for Arguments in Heterogeneous Sources (N18-5)

Challenge:	Recent efforts to improve the quality of machine-generated natural language content have been limited due to the large token usage required by complex evaluation prompts.
Approach:	They propose a prompt optimization approach that uses a smaller, fine-tuned language model to compress input data for evaluation prompt, thus reducing token usage and computational cost when using larger LLMs for downstream evaluation.
Outcome:	The proposed approach reduces token usage and costs by 2.37 compared with larger LLMs for downstream evaluation.

Copied to clipboard

Christian Stab, Johannes Daxenberger, Chris Stahlhut, Tristan Miller, Benjamin Schiller, Christopher Tauchmann, Steffen Eger, Iryna Gurevych

Information Retrieval

Challenge:	Argument mining is a core technology for enabling argument search in large corpora . but current methods fail when applied to heterogeneous texts . despite its obvious applications, argument search has attracted relatively little attention .
Approach:	They propose a system that searches sentential arguments for any given topic . ArgumenText automatically identifies and classifies arguments by relevance .
Outcome:	The proposed system covers 89% of arguments found in expert-curated lists . it also identifies additional valid arguments omitted or overlooked by human curators .

How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs (2025.naacl-long)

Copied to clipboard

Ran Zhang, Wei Zhao, Steffen Eger

EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics (2023.findings-emnlp)

Challenge:	Recent research has focused on literary machine translation (MT) but evaluation of literary MT remains an open problem.
Approach:	They propose a paragraph-level parallel corpus containing verified human translations and 13k evaluated sentences across four language pairs.
Outcome:	The proposed corpus compares human evaluations with students and professionals . it shows that the adequacy of human evaluation is controlled by two factors .

Copied to clipboard

Daniil Larionov, Jens Grünwald, Christoph Leiter, Steffen Eger

Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation (2026.findings-acl)

Challenge:	a recent surge of interest in developing evaluation metrics based on pretrained large language models (LLMs) can better cope with lexical variation.
Approach:	They propose to replace computation-intensive transformers with lighter alternatives and employ linear and quadratic approximations for alignment algorithms on top of LLM representations.
Outcome:	The proposed approach replaces computation-intensive transformers with lighter alternatives and employs linear and quadratic approximations for alignment algorithms on top of LLM representations.

Copied to clipboard

Ran Zhang, Steffen Eger, Arda Tezcan, Wei Zhao, Simone Paolo Ponzetto, Lieve Macken

Graph-Guided Textual Explanation Generation Framework (2025.emnlp-main)

Challenge:	Large language models (LLMs) are increasingly used for creative tasks such as literary translation.
Approach:	They propose a paired-task framework that assesses translational creativity using Units of Creative Potential (UCPs) they benchmark 23 models and four creativity-oriented prompts to assess translational comprehension .
Outcome:	The proposed framework compares 23 models and four creativity-oriented prompts on literary excerpts from 11 books.

Copied to clipboard

Shuzhou Yuan, Jingyi Sun, Ran Zhang, Michael Färber, Steffen Eger, Pepa Atanasova, Isabelle Augenstein

BMX: Boosting Natural Language Generation Metrics with Explainability (2024.findings-eacl)

Challenge:	Existing work has questioned their faithfulness, as they may not accurately reflect the model’s internal reasoning process regarding its predicted answer.
Approach:	They propose a Graph-Guided Textual Explanation Generation framework that generates a graph neural network layer that guides the NLE generation and generates explanations with greater semantic and lexical similarity to human-written ones.
Outcome:	The proposed framework improves NLE faithfulness by up to 12.12% compared to baseline methods on encoder-decoder and decoder-only models.

Copied to clipboard

Christoph Leiter, Hoa Nguyen, Steffen Eger

Does My Rebuttal Matter? Insights from a Major NLP Conference (N19-1)

Challenge:	Modern language model (LM) based natural language generation evaluation metrics achieve astonishing results in grading machine generated sentences like humans would.
Approach:	They use word-level explanations to convert a word- level score into a segment-level score and combine this segment- level with the original metric to obtain a better metric.
Outcome:	The proposed metrics improve across MT and summarization datasets while achieving small improvements on machine translation.

Copied to clipboard

Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, Yusuke Miyao

Responsible & Trustworthy NLP

Challenge:	Peer review is a core element of the scientific process, but few studies have evaluated its properties empirically.
Approach:	They propose to use peer review to assess the effectiveness of rebuttal phase in NLP conferences.
Outcome:	The proposed task predicts after-rebuttal scores from initial reviews and author responses.

Emotionally Charged, Logically Blurred: AI-driven Emotional Framing Impairs Human Fallacy Detection (2026.eacl-long)

Copied to clipboard

Yanran Chen, Lynn Greschner, Roman Klinger, Michael Klenk, Steffen Eger

Challenge:	Existing computational approaches focus on logical structures of fallacies and argumentation schemes, ignoring the emotional dimension of argumentation.
Approach:	They propose to use large language models to systematically change emotional appeals in fallacious arguments by using a computational approach.
Outcome:	The proposed method reduces fallacy detection by 14.5% on average on human arguments with enjoyment over fear or sadness.

Multi-Task Learning for Argumentation Mining in Low-Resource Settings (N18-2)

Copied to clipboard

Claudia Schulz, Steffen Eger, Johannes Daxenberger, Tobias Kahse, Iryna Gurevych

Low-Resource NLP

Challenge:	Argument component identification is difficult for trained annotators to perform in a new domain or to develop new AM tasks.
Approach:	They investigate whether multi-task learning can improve performance on AM problems . they found that MTL performs particularly well when little training data is available for the main task .
Outcome:	The proposed approach performs better when little training data is available for the main task, a common scenario in AM.

Trade-Offs Between Fairness and Privacy in Language Modeling (2023.findings-acl)

Copied to clipboard

Cleo Matzken, Steffen Eger, Ivan Habernal

SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020.acl-main)

Challenge:	Existing research suggests that privacy preservation comes at the price of worsening biases in classification tasks.
Approach:	They propose to incorporate privacy preservation and de-biasing techniques into training text generation models to investigate the trade-off between the two dimensions.
Outcome:	The proposed model improves on bias detection, privacy attacks, language modeling, and performance on downstream tasks.

Copied to clipboard

Yang Gao, Wei Zhao, Steffen Eger

Low-Resource NLP

Challenge:	Existing evaluation methods for document summarization require human annotations and annotations.
Approach:	They propose a method which measures the quality of a summary by measuring its semantic similarity with a pseudo reference summary, using contextualized embeddings and soft token alignment techniques.
Outcome:	The proposed method correlates better with human ratings by 18- 39% compared to the state-of-the-art evaluation metrics.

Do Emotions Really Affect Argument Convincingness? A Dynamic Approach with LLM-based Manipulation Checks (2025.findings-acl)

Copied to clipboard

Yanran Chen, Steffen Eger

Challenge:	Emotions have been shown to play a role in argument convincingness, yet this aspect is underexplored in the natural language processing community.
Approach:	They propose a framework that examines the extent to which perceived emotional intensity influences perceived convincingness.
Outcome:	The proposed framework examines whether emotions influence persuasiveness in humans . it finds that emotions enhance rather than weaken convincingness in human judgments .

Inducing Language-Agnostic Multilingual Representations (2021.starsem-1)

Copied to clipboard

Wei Zhao, Steffen Eger, Johannes Bjerva, Isabelle Augenstein

ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models (2023.acl-long)

Challenge:	Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world, but they currently require large pretraining corpora or access to typologically similar languages.
Approach:	They propose to remove language identity signals from multilingual embeddings by re-aligning vector spaces of target languages to a pivot source language and removing language-specific means and variances.
Outcome:	The proposed approaches reduce cross-lingual transfer gap by 8.9 points (m-BERT) and 18.2 points (XLM-R) on average across all tasks and languages.

Copied to clipboard

Jonas Belouadi, Steffen Eger

UScore: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation (2023.eacl-main)

Challenge:	End-to-end models learn to complete a task by directly learning all steps, without intermediary algorithms such as hand-crafted rules or post-processing.
Approach:	They propose to train end-to-end poetry generation conditioned on styles such as rhyme, meter, and alliteration . they pre-train ByGPT5, a new token-free decoder-only language model, and fine-tune it on a custom corpus of English and German quatrains .
Outcome:	The proposed model outperforms other models on a large custom corpus of English and German quatrains while being more parameter efficient and performing favorably compared to humans.

Copied to clipboard

Jonas Belouadi, Steffen Eger

Evaluating Diversity in Automatic Poetry Generation (2024.emnlp-main)

Challenge:	supervised evaluation metrics are not available for machine translation, despite their wide dissemination.
Approach:	They develop fully unsupervised evaluation metrics that leverage parallel data and evaluation metric induction.
Outcome:	The proposed metrics beat supervised competitors on 4 out of 5 evaluation datasets.

Copied to clipboard

Yanran Chen, Hannes Gröner, Sina Zarrieß, Steffen Eger

From Hero to Zéroe: A Benchmark of Low-Level Adversarial Attacks (2020.aacl-main)

Challenge:	Existing models for creative text generation are not evaluated regarding how different generated poems are from existing training sets.
Approach:	They evaluate the diversity of automatically generated poetry by comparing distributions of generated poetry to distributions in human poetry along structural, lexical, semantic and stylistic dimensions.
Outcome:	The proposed model types show that style-conditioning and character-level modeling increases diversity across virtually all dimensions.

Copied to clipboard

Steffen Eger, Yannik Benz

Robustness in NLP

Challenge:	Adversarial attacks are label-preserving modifications to inputs of machine learning classifiers designed to fool machines but not humans.
Approach:	They propose to use a dataset to test the robustness of future NLP models to identify low-level adversarial attacks that are less realistic in typical applications such as social media.
Outcome:	The proposed dataset provides a benchmark for testing robustness of future more human-like NLP models.

PO-EMO: Conceptualization, Annotation, and Modeling of Aesthetic Emotions in German and English Poetry (2020.lrec-1)

Copied to clipboard

Thomas Haider, Steffen Eger, Evgeny Kim, Roman Klinger, Winfried Menninghaus

Sentiment Analysis

Challenge:	a new study shows that literature enables engagement in a broader range of complex and subtle emotions.
Approach:	They propose to use multiple emotion labels to capture mixed emotions in poetry . they evaluate an annotation experiment with experts and crowdsourcing .
Outcome:	The proposed method shows that identifying aesthetic emotions is challenging in the German subset.

Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021.emnlp-main)

Copied to clipboard

Marvin Kaster, Wei Zhao, Steffen Eger

Explainability & Interpretability in NLP

Challenge:	Evaluation metrics are a key ingredient for progress of text generation systems . a class of novel evaluation metrics based on BERT and its variants has been explored .
Approach:	They propose to disentangle BERT-based evaluation metrics along linguistic factors . they show they are sensitive to lexical overlap, just like BLEU and ROUGE .
Outcome:	The proposed metrics capture all aspects but are sensitive to lexical overlap, just like BLEU and ROUGE, the authors show .

BERT-Defense: A Probabilistic Model Based on BERT to Combat Cognitively Inspired Orthographic Adversarial Attacks (2021.findings-acl)

Copied to clipboard

Yannik Keller, Jan Mackensen, Steffen Eger

Robustness in NLP

Challenge:	adversarial attacks expose important blind spots of deep learning systems, authors show . word and sentence-level attacks tend to be more difficult to defend via spelling correction modules . character-level attack scenarios often involve finding semantic paraphrases of input .
Approach:	They propose a model that probabilistically combines context-independent word level information with context-dependent information from BERT's masked language modeling to combat low-level orthographic attacks.
Outcome:	The proposed model outperforms a spellchecker and Pruthi's model on a character-level benchmark.

On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation (2020.acl-main)

Copied to clipboard

Wei Zhao, Goran Glavaš, Maxime Peyrard, Yang Gao, Robert West, Steffen Eger

Killing Four Birds with Two Stones: Multi-Task Learning for Non-Literal Language Detection (C18-1)

Challenge:	a standard evaluation setup for supervised machine learning tasks does not hold for natural language generation tasks.
Approach:	They propose to use reference-free machine translation evaluation to compare source texts to system translations to find key limitations.
Outcome:	The proposed metrics perform poorly as semantic encoders for reference-free machine translation evaluation.

Copied to clipboard

Erik-Lân Do Dinh, Steffen Eger, Iryna Gurevych

GerAV: Towards New Heights in German Authorship Verification using Fine-Tuned LLMs on a New Benchmark (2026.findings-acl)

Challenge:	idioms and metaphors are often studied in isolation, challenging the distinction . e.g., metaphorical concept mappings are ubiquitous in everyday life, thus they are ubiquitous .
Approach:	They propose to view the detection problem as a generalized non-literal language classification problem.
Outcome:	The proposed model improves on four metaphor and idiom detection tasks in two languages, English and German.

Copied to clipboard

Lotta Kiefer, Christoph Leiter, Sotaro Takeshita, Elena Schmidt, Steffen Eger

Changes in European Solidarity Before and During COVID-19: Evidence from a Large Crowd- and Expert-Annotated Twitter Dataset (2021.acl-long)

Challenge:	Authorship verification (AV) is a task of determining whether two texts were written by the same author.
Approach:	They propose a benchmark for German AV comprising over 400k labeled text pairs.
Outcome:	The proposed model outperforms baselines and state-of-the-art models by 0.09 and surpasses GPT-5 in a zero-shot setting by 0.08.

Copied to clipboard

Alexandra Ils, Dan Liu, Daniela Grunow, Steffen Eger

Probing Multilingual BERT for Genetic and Typological Signals (2020.coling-main)

Challenge:	Using supervised machine learning, we assess how solidarity discourses changed before and during the COVID-19 crisis.
Approach:	They use social scientific concept of solidarity and its contestation, anti-solidarity, as problem setting to assess how European solidarity discourses changed before and during COVID-19.
Outcome:	The proposed model outperforms the baseline classifier with expert annotations by 25 points, from 58% macro-F1 to almost 85%.

Copied to clipboard

Taraka Rama, Lisa Beinborn, Steffen Eger

Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems (N19-1)

Challenge:	Recent cross-lingual models provide representations for about 100 languages and vary in their training objectives.
Approach:	They probe the layers in multilingual BERT for phylogenetic and geographic language signals across 100 languages and compute language distances based on the mBERT representations.
Outcome:	The proposed model is best explained by phylogenetic and worst by structural factors and correlates with published ranked lists based on linguistic approaches.

Copied to clipboard

Steffen Eger, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, Iryna Gurevych

Robustness in NLP

Challenge:	Recent studies show that visual similarity can play a decisive role in assessing the meaning of characters.
Approach:	They investigate the impact of visual adversarial attacks on current NLP systems . they explore three shielding methods that significantly improve the robustness of the models .
Outcome:	The proposed methods improve performance but still fall behind non-attack scenarios.

MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (D19-1)

Copied to clipboard

Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, Steffen Eger

Representation Learning

Challenge:	Existing evaluation metrics are not capable of evaluating text quality.
Approach:	They propose a metric that compares system output against reference texts based on semantics rather than surface forms.
Outcome:	The proposed metric shows a high correlation with human judgment of text quality on a number of text generation tasks.

Cross-lingual Argumentation Mining: Machine Translation (and a bit of Projection) is All You Need! (C18-1)

Copied to clipboard

Steffen Eger, Johannes Daxenberger, Christian Stab, Iryna Gurevych

Dependencies over Times and Tools (DoTT) (2024.lrec-main)

Challenge:	Argumentation mining (AM) requires the identification of complex discourse structures . existing resources are not adequate for assessing cross-lingual AM due to their heterogeneity or lack of complexity.
Approach:	They propose to use a dataset to translate persuasive student essays into German, French, Spanish, and Chinese to compare arguments mining and annotation projection.
Outcome:	The proposed methods perform better when using expensive human or cheap machine translations and almost eliminate loss from cross-lingual transfer.

Copied to clipboard

Andy Luecking, Giuseppe Abrami, Leon Hammerla, Marc Rahn, Daniel Baumartz, Steffen Eger, Alexander Mehler

Challenge:	Using the examples of English and German, we examine how parsers trained on modern variants of these languages can be transferred to older language levels without loss.
Approach:	They develop a treebank of diachronic corpora enriched with dependency annotations using 3 parsers, 6 pre-trained language models, 5 newly trained models for German, and two tag sets.
Outcome:	The proposed treebank covers the time period from 1800 until today and is based on the DependencyAnnotator annotation tool.

xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics (2024.emnlp-main)

Copied to clipboard

Daniil Larionov, Mikhail Seleznyov, Vasiliy Viskov, Alexander Panchenko, Steffen Eger

Argument Summarization and its Evaluation in the Era of Large Language Models (2025.emnlp-main)

Challenge:	State-of-the-art trainable machine translation evaluation metrics rely on large encoders . this makes them computationally expensive and inaccessible to researchers with limited resources.
Approach:	They propose a method to extract knowledge stored in large encoders and a pipeline for efficient black-box distillation.
Outcome:	The proposed model surpasses COMET-22 and BLEURT-20 on the WMT22 dataset by 6.4%.

Copied to clipboard

Moritz Altemeyer, Steffen Eger, Johannes Daxenberger, Yanran Chen, Tim Altendorf, Philipp Cimiano, Benjamin Schiller

Reproducibility Issues for BERT-based Evaluation Metrics (2022.emnlp-main)

Challenge:	Large Language Models (LLMs) have revolutionized various Natural Language Generation tasks, including Argument Summarization (ArgSum).
Approach:	They propose a prompt-based evaluation scheme and validate it through a human benchmark dataset.
Outcome:	The proposed evaluation scheme outperforms existing methods and is validated by a human benchmark dataset.

Copied to clipboard

Yanran Chen, Jonas Belouadi, Steffen Eger

Layer or Representation Space: What Makes BERT-based Evaluation Metrics Robust? (2022.coling-1)

Challenge:	Reproducibility is of utmost concern in machine learning and natural language processing . lexical-overlap metrics are still the dominant metric in natural language generation .
Approach:	They ask whether results and claims from four recent BERT-based evaluation metrics can be reproduced.
Outcome:	The proposed metrics outperform the dominant metric, BLEU, and show that they can be reproduced.

Copied to clipboard

Doan Nam Long Vu, Nafise Sadat Moosavi, Steffen Eger

PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation (2024.emnlp-main)

Challenge:	Recent embedding-based evaluation metrics for text generation are based on measuring correlation with human evaluations on standard benchmarks.
Approach:	They examine the robustness of BERTScore, one of the most popular embedding-based metrics for text generation.
Outcome:	The embedding-based metrics that have the highest correlation with human evaluations on a standard benchmark can have the lowest correlation if the amount of input noise or unknown tokens increases.

Copied to clipboard

Christoph Leiter, Steffen Eger