Papers by Lukas Lange
Rehearsal-Free Modular and Compositional Continual Learning for Language Models (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing methods to overcome catastrophic forgetting are rehearsal-based and parameter isolation-based. |
| Approach: | They propose a rehearsal-free framework which continuously adds new modules to language models and composes them with existing modules. |
| Outcome: | Experiments on benchmarks show that MoCL outperforms state-of-the-art and effectively facilitates knowledge transfer. |
QUITE: Quantifying Uncertainty in Natural Language Text in Bayesian Reasoning Scenarios (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing probabilistic reasoning datasets require the model to only rank textual alternatives or use limited set of templates. |
| Approach: | They propose a question-answering dataset that uses probabilistic rules to express degrees of certainty. |
| Outcome: | The proposed model outperforms existing models on all reasoning types . it is available on Github and is expected to be used in clinical documentation . |
Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art methods for updating large language models require computational overhead and lack theoretical validation. |
| Approach: | They propose a model editing method that uses sentence concatenation with augmented random facts for generation regularization. |
| Outcome: | The proposed method outperforms state-of-the-art methods while maintaining generation quality and reducing computational overhead. |
Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Multilingual language models store factual knowledge across languages but struggle to provide consistent responses to semantically equivalent prompts in different languages. |
| Approach: | They propose a linear shortcut method that bypasses computations in the final layers . this method improves accuracy and cross-lingual consistency . |
| Outcome: | The proposed method improves prediction accuracy and cross-lingual consistency. |
Discourse-Aware In-Context Learning for Temporal Expression Normalization (2024.naacl-short)
Copied to clipboard
| Challenge: | Temporal expression (TE) normalization is a well-studied problem, but upcoming machine learning approaches suffer from a lack of labeled data. |
| Approach: | They propose to use in-context learning to inject task, document, and example information into a large language model for temporal expression normalization. |
| Outcome: | The proposed model performs better in non-standard settings by dynamically including relevant examples during inference. |
GradSim: Gradient-Based Language Grouping for Effective Multilingual Training (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that not all languages positively influence each other . multilingual training can help in those cases by sharing knowledge across languages . |
| Approach: | They propose a gradient similarity-based language grouping method for multilingual training that is better correlated with cross-lingual model performance. |
| Outcome: | The proposed method leads to the largest performance gains on a multilingual dataset and is better correlated with cross-lingual model performance. |
DelucionQA: Detecting Hallucinations in Domain-specific Question Answering (2023.findings-emnlp)
Copied to clipboard
Mobashir Sadat, Zhengyu Zhou, Lukas Lange, Jun Araki, Arsalan Gundroo, Bingqing Wang, Rakesh Menon, Md Parvez, Zhe Feng
| Challenge: | Hallucination is a well-known phenomenon in text generated by large language models . state-of-the-art LLMs still have a number of weaknesses, including the tendency to generate hallucinatory statements without considering the factuality . |
| Approach: | They propose a dataset that captures hallucinations made by retrieval-augmented LLMs . they propose to use these methods to help detect hallucinosity in QA tasks . |
| Outcome: | The proposed method captures hallucinations made by retrieval-augmented LLMs for QA tasks. |
AnnoCTR: A Dataset for Detecting and Linking Entities, Tactics, and Techniques in Cyber Threat Reports (2024.lrec-main)
Copied to clipboard
Lukas Lange, Marc Müller, Ghazaleh Haratinezhad Torbati, Dragan Milchevski, Patrick Grau, Subhash Chandra Pujari, Annemarie Friedrich
| Challenge: | Abstract: Natural language processing can help with managing large amounts of unstructured information. |
| Approach: | They propose to annotate a CC-BY-SA-licensed dataset of cyber threat reports . they use named entities, temporal expressions, and cybersecurity-specific concepts . |
| Outcome: | The proposed dataset annotates reports with named entities, temporal expressions, and cybersecurity-specific concepts including implicitly mentioned techniques and tactics. |
To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to select transfer sources are limited by text and task similarity, which limits their application in transfer settings where both the task and the text domain change. |
| Approach: | They propose a model similarity measure that represents text and task similarity jointly to automatically determine which and how many sources to exploit. |
| Outcome: | The proposed approach improves performance by 24 F1 points for predicting promising sources across domains and tasks with similar models. |
SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent work shows promising results when prompting pre-trained language models, but in low-resource domains, the domain gap between the pre-training data and the downstream task is too large. |
| Approach: | They propose a method for prompting pre-trained language models using domain-specific keywords with a trainable gated prompt. |
| Outcome: | The proposed prompting method outperforms state-of-the-art prompting methods on three text classification benchmarks and shows that it reduces the need for domain-specific language model pre-training. |
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2021.naacl-main)
Copied to clipboard
| Challenge: | a growing body of work is focused on improving performance in low-resource settings . a goal of this study is to explain how these methods differ in their requirements . |
| Approach: | They propose to analyze data-lean scenarios across different dimensions of data availability to understand which approaches are effective in a specific low-resource setting. |
| Outcome: | The proposed methods enable learning when training data is sparse. |
TADA: Efficient Task-Agnostic Domain Adaptation for Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained transformer-based language models are limited in their expressiveness and domain knowledge. |
| Approach: | They propose a task-agnostic domain adaptation method which is modular, parameter-efficient, and data-efficient. |
| Outcome: | The proposed method is efficient and modular, parameter-efficient, and data-efficient. |
FAME: Feature-Based Adversarial Meta-Embeddings for Robust Input Representations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on word embeddings and pre-trained language models has shown the large impact of language representations on natural language processing (NLP) models across tasks and domains. |
| Approach: | They propose feature-based adversarial meta-embeddings with an attention function that is guided by word-specific properties, such as shape and frequency, to handle subword-based embeddings. |
| Outcome: | The proposed model improves performance in downstream tasks even with word embeddings from transformers. |
NLNDE: Enhancing Neural Sequence Taggers with Attention and Noisy Channel for Robust Pharmacological Entity Detection (D19-57)
Copied to clipboard
| Challenge: | Named entity recognition has been extensively studied on English news texts, but transfer to other domains and languages is still a challenging problem. |
| Approach: | They propose a system that provides a non-standard domain and language setting for pharmacological entity detection in Spanish texts and a sequencelabeling task that requires neither language nor domain expertise. |
| Outcome: | The proposed system achieves up to 88.6% F1 in the PharmaCoNER competition and is based on a sequence labeling task and training on annotated data. |
Feature-Dependent Confusion Matrices for Low-Resource NER Labeling with Noisy Labels (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to improve supervised labeling with noisy training data do not take the input features into account or they need to learn the noise modeling from scratch. |
| Approach: | They propose to cluster training data using input features and compute different confusion matrices for each cluster. |
| Outcome: | The proposed model improves on low-resource named entity recognition settings in several languages, compared with other models which do not take the input features into account or need to learn noise modeling from scratch. |
Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Reasoning language models (RLMs) excel at complex tasks by leveraging a chain-of-thought process to generate structured intermediate steps. |
| Approach: | They present the first systematic study of language mixing in reasoning language models, examining its patterns, impact, and internal causes across 15 languages, 7 task difficulty levels, and 18 subject areas. |
| Outcome: | The proposed model generates reasoning steps that include a mixture of languages when prompted in one language, and this improves accuracy. |
Closing the Gap: Joint De-Identification and Concept Extraction in the Clinical Domain (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies show that de-identification is effective in the clinical domain but not in the downstream tasks. |
| Approach: | They propose a stacked model with restricted access to privacy sensitive information and a multitask model to investigate the effect of de-identification on clinical concept extraction. |
| Outcome: | The proposed model is stacked with restricted access to privacy sensitive information and a multitask model. |
KRAUTS: A German Temporally Annotated News Corpus (L18-1)
Copied to clipboard
| Challenge: | Temporal tagging is an important task towards improved natural language understanding. |
| Approach: | They present a new German temporally annotated corpus with 192 documents with 1,140 annotations . they propose to make temporal tagging a viable research area . |
| Outcome: | The proposed corpus contains 192 documents with 1,140 annotated temporal expressions. |
Multilingual Normalization of Temporal Expressions with Masked Language Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for normalizing temporal expressions are rule-based, which severely limits the applicability in multilingual settings. |
| Approach: | They propose a neural method for normalizing temporal expressions based on masked language modeling and a slot-based prediction scheme for context-independent representations. |
| Outcome: | The proposed method outperforms existing rule-based methods in many languages and in particular, for low-resource languages with performance improvements of up to 33 F1 on average compared to the state of the art. |
The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain (2020.acl-main)
Copied to clipboard
Annemarie Friedrich, Heike Adel, Federico Tomazic, Johannes Hingerl, Renou Benteau, Anika Marusczyk, Lukas Lange
| Challenge: | Using BERT embeddings leads to large performance gains, but with increasing task complexity, adding a recurrent neural network seems beneficial. |
| Approach: | They propose an annotation scheme for marking information on publications related to solid oxide fuel cells . they propose to use a recurrent neural network to solve a variety of tasks . |
| Outcome: | The proposed scheme is based on a corpus of 45 open-access scholarly articles and a neural network for a variety of tasks. |