Papers by Florian Matthes
DA-Pred: Performance Prediction for Text Summarization under Domain-Shift and Instruct-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often don’t perform as expected under Domain Shift or after Instruct-tuning. |
| Approach: | They propose a method that uses the known performance in high-resource domains and fine-tuning settings to predict performance in low-resourced domains or base models. |
| Outcome: | The proposed method can help researchers decide if resources should be allocated for data labeling and LLM Instruct-tuning. |
Thinking Outside of the Differential Privacy Box: A Case Study in Text Privatization with Language Model Prompting (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on the integration of Differential Privacy (DP) into NLP techniques. |
| Approach: | They propose a method for text privatization leveraging language models to rewrite texts . they examine the usability of DP in NLP and its benefits over non-DP approaches . |
| Outcome: | The proposed method is a novel method for text privatization leveraging language models to rewrite texts. |
Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLMs encode extensive knowledge within their parameters, but the knowledge in LLM models can become outdated over time. |
| Approach: | They propose two new LLMs that provide outdated medical advice . they compare the models with a set of QA pairs whose verdict changed through time . |
| Outcome: | The proposed models exhibit memorization of outdated knowledge to some extent. |
Comparing Knowledge Sources for Open-Domain Scientific Claim Verification (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing systems for fact-checking scientific claims assume that the documents containing the evidence are already provided and annotated or contained in a limited corpus. |
| Approach: | They perform an array of experiments to test the performance of open-domain claim verification systems on four datasets of biomedical and health claims in different settings. |
| Outcome: | The proposed system performs better with biomedical and health claims, while Wikipedia is more suited for everyday health concerns. |
CarMem: Enhancing Long-Term Memory in LLM Voice Assistants through Category-Bounding (2025.coling-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are stateless and present all relevant memories during each interaction, resulting in repetitive user requests and disengagement. |
| Approach: | They propose a long-term memory system for voice assistants structured around predefined categories that leverages Large Language Models to extract, store, and retrieve preferences within these categories. |
| Outcome: | The proposed system achieves an F1-score of .78 to .95 in preference extraction, depending on category granularity, and is suitable for industrial applications. |
A Decade of Knowledge Graphs in Natural Language Processing: A Survey (2022.aacl-main)
Copied to clipboard
| Challenge: | Knowledge graphs (KGs) are a representation of semantic relations between entities . despite their popularity, there is still no general understanding of what exactly a KG is or for what tasks it is applicable. |
| Approach: | They analyze 507 papers on knowledge graphs in natural language processing (NLP) they provide a taxonomy of tasks and review the maturity of individual research streams . |
| Outcome: | The findings summarize the literature and highlight directions for future work. |
Text2Tabular – Reconstructing Tabular Research Data from Scientific Publications (2026.acl-long)
Copied to clipboard
| Challenge: | Text2Tabular reconstructs research datasets from scientific literature using advanced natural language processing and statistical modeling. |
| Approach: | Text2Tabular reconstructs research datasets from scientific publications using natural language processing and statistical modeling. |
| Outcome: | Text2Tabular reconstructs scientific literature-based datasets using natural language processing and statistical modeling. |
MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models can encode knowledge during pre-training on large text corpora, enabling downstream tasks like question answering (QA). |
| Approach: | They construct a dataset derived from systematic reviews to examine their ability to encode medical knowledge and their recall. |
| Outcome: | The proposed model performs well on the biomedical QA dataset. |
Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion (2025.findings-acl)
Copied to clipboard
| Challenge: | Using later reasoning steps does not always improve classification, suggesting LLMs encode key information early. |
| Approach: | They propose a method to predict the success of a zero-shot Chain-of-Thought process by using LLM representations that are based on initial steps representations. |
| Outcome: | The proposed method performs well even before a single token is generated, suggesting that crucial information about the reasoning process is already present in the initial steps representations. |
On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems (2025.findings-naacl)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) is an approach to augment large language models (LLMs) despite their impressive performance, LLMs can generate plausible sounding but factually incorrect responses (hallucinations) |
| Approach: | They propose to use BM25 and semantic search as retrievers to augment large language models by reducing their reliance on static knowledge and improving answer factuality. |
| Outcome: | The proposed approach improves QA performance on a biomedical task with up to 15 snippets but stagnates or declines beyond that. |
On the Impact of Noise in Differentially Private Text Rewriting (2025.findings-naacl)
Copied to clipboard
| Challenge: | a field of text privatization often requires the addition of noise to vector representations of text . however, noise addition leads to considerable utility loss, highlighting one drawback of DP in NLP. |
| Approach: | They propose a sentence-infilling privatization technique that adds noise to vector representations of text to provide privacy guarantees. |
| Outcome: | The proposed method shows that non-DP privatization methods excel in utility preservation and can find an acceptable privacy-utility trade-off, but cannot outperform DP methods in empirical privacy protections. |
Improving Health Question Answering with Reliable and Time-Aware Evidence Retrieval (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing question answering systems rely on pre-selected and annotated evidence documents, thus making them inadequate for addressing novel questions. |
| Approach: | They propose to use the common retrieve-then-read QA pipeline and PubMed as a trustworthy collection of medical research documents to answer health questions from three diverse datasets. |
| Outcome: | The proposed approach improves the macro F1 score by 10% by utilizing the common retrieve-then-read QA pipeline and PubMed as a trustworthy collection of medical research documents. |
Beyond Grid Search: Leveraging Bayesian Optimization for Accelerating RAG Pipeline Optimization (2026.eacl-industry)
Copied to clipboard
| Challenge: | Finding optimal configurations via grid search is computationally prohibitive, limiting real-world scalability. |
| Approach: | They compare BO with grid search to find optimal configurations for RAG pipelines . their framework explores global pipeline and local component-wise optimization . |
| Outcome: | The proposed approach reduces optimization time by up to 84% while maintaining comparable accuracy. |
A Comparative Analysis of Word-Level Metric Differential Privacy: Benchmarking the Privacy-Utility Trade-off (2024.lrec-main)
Copied to clipboard
| Challenge: | Differential Privacy (DP) has been used in NLP for years to address privacy concerns . privacy-enhancing technologies (PETs) are concrete technical solutions that can be incorporated into existing systems. |
| Approach: | They compare different Differential Privacy algorithms for word-level NLP tasks . they propose concrete steps forward to combat privacy risks in NLP settings . |
| Outcome: | The proposed methods perform better than the proposed methods on two NLP tasks. |
AGB-DE: A Corpus for the Automated Legal Assessment of Clauses in German Consumer Contracts (2024.acl-long)
Copied to clipboard
| Challenge: | Annotated contracts are laborious task performed by companies, law firms, NGOs and the scientific community. |
| Approach: | They present a corpus of 3,764 clauses from German consumer contracts annotated by legal experts with a clause in the contract. |
| Outcome: | The proposed framework outperforms openly available models in detecting potentially void clauses. |
A Comparative Analysis of Conversational Large Language Models in Knowledge-Based Text Generation (2024.eacl-short)
Copied to clipboard
| Challenge: | Generating natural language text from graph-structured data is essential for conversational information seeking. |
| Approach: | They conduct an empirical analysis of conversational large language models in generating natural language text from semantic triples using a WebNLG dataset. |
| Outcome: | The proposed models improve their ability to generate natural language text from semantic triples using few-shot prompting, post-processing, and efficient fine-tuning techniques. |
HealthFC: Verifying Health Claims with Evidence-Based Medical Fact-Checking (2024.lrec-main)
Copied to clipboard
| Challenge: | determining the trustworthiness of online medical content is challenging in the digital age . fact-checking is an approach to assess the veracity of factual claims . a new dataset is presented to help advance automated fact- checking . |
| Approach: | They propose a dataset that assesses the veracity of factual claims using evidence from credible sources. |
| Outcome: | The proposed dataset can be used for automated fact-checking tasks. |
NLP-KG: A System for Exploratory Search of Scientific Literature in Natural Language Processing (2024.acl-demos)
Copied to clipboard
| Challenge: | Existing systems for scientific literature search are typically tailored to keyword-based lookup searches, limiting possibilities for exploration. |
| Approach: | They propose a feature-rich system that supports the exploration of research literature in unfamiliar natural language processing fields. |
| Outcome: | The proposed system supports the exploration of research literature in unfamiliar natural language processing fields. |
Step-by-Step Fact Verification System for Medical Claims with Explainable Reasoning (2025.naacl-short)
Copied to clipboard
| Challenge: | Fact verification (FV) aims to assess the veracity of a claim based on relevant evidence. |
| Approach: | They propose to use iterative fact verification to assess the veracity of a claim based on relevant evidence. |
| Outcome: | The proposed system improves on three medical fact-checking datasets and evaluates with multiple settings including different LLMs, external web search, and structured reasoning using logic predicates. |
Scientific Fact-Checking: A Survey of Resources and Approaches (2023.findings-acl)
Copied to clipboard
| Challenge: | Fact-checking is the task of assessing the veracity of factual claims based on credible evidence and background knowledge. |
| Approach: | They propose to automate scientific fact-checking using natural language processing to assess the veracity of factual claims based on credible evidence and background knowledge. |
| Outcome: | The proposed methods can help combat the spread of misinformation and help individuals understand new scientific breakthroughs. |
DP-MLM: Differentially Private Text Rewriting Using Masked Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for text privatization using Differential Privacy rely on autoregressive models which lack a mechanism to contextualize the private rewriting process. |
| Approach: | They propose a method for differentially private text rewriting using masked language models to rewrite a text one token at a time. |
| Outcome: | The proposed method preserves utility at lower levels, compared to previous methods relying on autoregressive models with a decoder. |
Leveraging Semantic Triples for Private Document Generation with Local Differential Privacy Guarantees (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on differential privacy in text privatization use word perturbations and rewriting methods to protect privacy. |
| Approach: | They propose a method which leverages semantic triples for neighborhood-aware private document generation under local DP guarantees. |
| Outcome: | The proposed method allows for coherent text generation even at lower values while still balancing privacy and utility. |
MucLex: A German Lexicon for Surface Realisation (2020.lrec-1)
Copied to clipboard
| Challenge: | lexica for rule-based surface realisers for languages other than English are often scarce. |
| Approach: | They propose a German lexicon for the Natural Language Generation task of surface realisation based on the crowd-sourced online lexicont Wiktionary. |
| Outcome: | The proposed lexicon contains more than 100,000 lemmata and 670,000 different word forms. |
Verifiable Parameterization of Bayesian Networks from Scientific Literature: Unlocking Unstructured Empirical Evidence (2026.findings-acl)
Copied to clipboard
| Challenge: | Current methods to learn conditional probabilities from raw tabular data are limited due to privacy concerns or general lack of access to data. |
| Approach: | They propose to reconstruct local conditional probability tables solely from statistical summaries to parameterize Bayesian Networks. |
| Outcome: | The proposed methods outperform baseline methods while ensuring transparency and verifiability. |