Papers by Dennis Aumiller
EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization datasets focus on overly exposed domains and are primarily monolingual with few multilingual datasets. |
| Approach: | They propose a new summarization dataset based on manually curated document summaries from the European Union law platform EUR-Lex. |
| Outcome: | The proposed dataset is based on document summaries of legal acts from the European Union law platform (EUR-Lex). |
Klexikon: A German Dataset for Joint Summarization and Simplification (2022.lrec-1)
Copied to clipboard
| Challenge: | Traditionally, Text Simplification is a monolingual translation task where individual sentences are "translated" into a simplified version. |
| Approach: | They propose to use a dataset to jointly simplify long source documents by combining sentences from a source and their simplified counterparts. |
| Outcome: | The proposed system can summarize and simplify long source documents using almost 2,900 documents. |
How Does Quantization Affect Multilingual LLMs? (2024.findings-emnlp)
Copied to clipboard
Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis Aumiller, Ahmet Üstün, Sara Hooker, Sebastian Ruder
| Challenge: | Quantization is widely used to improve inference speed and deployment of large language models. |
| Approach: | They conduct a thorough analysis of quantized multilingual LLMs . they find language disparately affected by quantization, non-Latin script languages worst . authors urge consideration of multilingual performance as evaluation criterion for efficient models . |
| Outcome: | The results show that quantization has harmful effects on human evaluation . language performance is disparately affected by quantization, the authors say . |
Check Your Work: Structured Checklist Feedback for Improving Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have been driven by verifiable feedback in deterministic domains like mathematics and code. |
| Approach: | They propose to decompose granular, prompt-specific checklists into a scalar reward and use them to transform them into skalar rewards. |
| Outcome: | The proposed approach yields an 11.8% win-rate improvement on AlpacaEval 2.0 using Qwen3-8B, outperforming holistic reward models and existing checklist baselines. |
Evaluating Factual Consistency of Texts with Semantic Role Labeling (2023.starsem-1)
Copied to clipboard
| Challenge: | Existing evaluation methods rely on task-specific language models, which in turn hampers interpretation of generated scores. |
| Approach: | They propose a reference-free evaluation metric for text summarization that measures factuality . their method generates fact tuples from Semantic Role Labels, applied to both input and summary texts. |
| Outcome: | The proposed evaluation metric is comparable with state-of-the-art methods and has a stable generalization across datasets. |
On Leakage of Code Generation Evaluation Datasets (2024.findings-emnlp)
Copied to clipboard
Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon, Matthias Gallé
| Challenge: | In this paper, we discuss contamination by code generation test sets in large language models. |
| Approach: | They propose to use Python to test code generation test sets for contamination . they find that code generation is an important skill for large language models to master . |
| Outcome: | The proposed benchmarks are uncontaminated and provide a new insight into code generation. |
BLESS: Benchmarking Large Language Models on Sentence Simplification (2023.emnlp-main)
Copied to clipboard
Tannon Kew, Alison Chi, Laura Vásquez-Rodríguez, Sweta Agrawal, Dennis Aumiller, Fernando Alva-Manchego, Matthew Shardlow
| Challenge: | BLESS is a performance benchmark of the most recent state-of-the-art Large Language Models (LLMs) on the task of text simplification (TS). |
| Approach: | They present a performance benchmark of the most recent state-of-the-art Large Language Models (LLMs) on the task of text simplification (TS). |
| Outcome: | The proposed benchmarks show that the most recent state-of-the-art LLMs perform better on the task of text simplification (TS). |