Papers by Carolin Lawrence
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions (2025.emnlp-main)
Copied to clipboard
Emmy Liu, Amanda Bertsch, Lintang Sutawika, Lindia Tjuatja, Patrick Fernandes, Lara Marinov, Michael Chen, Shreya Singhal, Carolin Lawrence, Aditi Raghunathan, Kiril Gashteovski, Graham Neubig
| Challenge: | Language model performance is largely dependent on pretraining decisions, but scaling laws based on only these two aspects do not always explain downstream task performance. |
| Approach: | They meta-analyze 92 open-source pretrained models to quantify their impact on performance. |
| Outcome: | The framework lays a foundation for more systematic investigation of how model development choices shape final capabilities. |
Evaluating Language Models as Synthetic Data Generators (2025.acl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, Graham Neubig
| Challenge: | Prior studies have focused on developing effective data generation methods, but lack systematic comparison of different LMs as data generators in a unified setting. |
| Approach: | They propose to use a benchmark to compare language models' data generation abilities against a set of standardized settings and metrics. |
| Outcome: | The proposed benchmark provides standardized settings and metrics to evaluate LMs’ data generation abilities. |
Scaling Evaluation-Time Compute with Reasoning Models as Evaluators (2026.findings-acl)
Copied to clipboard
Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Minkyeong Moon, Carolin Lawrence, Kiril Gashteovski, Julia Hockenmaier, Graham Neubig, Sean Welleck
| Challenge: | Language model (LM) evaluators that generate chain-of-thought reasoning are widely used for the assessment of LM responses. |
| Approach: | They investigate whether increasing LMs' "thinking" time through scaling test-time compute can improve an LM's evaluation capability. |
| Outcome: | The proposed reasoning models improve evaluation performance monotonically with the number of reasoning tokens generated, mirroring trends seen in LM reasoning. |
BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for OIE are incomplete and do not include all acceptable variants of the same fact. |
| Approach: | They introduce BenchIE: a benchmark and evaluation framework for comprehensive evaluation of OIE systems for English, Chinese, and German. |
| Outcome: | The proposed framework is based on fact synsets, clusters, and standardized benchmarks. |
Linking Surface Facts to Large-Scale Knowledge Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Open Information Extraction (OIE) methods extract facts in the form of triples . ambiguity of these triples hinders their downstream usage . |
| Approach: | They propose a benchmark that measures fact linking performance on a granular triple slot level . they propose to use a system that can detect out-of-KG entities and predicates . |
| Outcome: | The proposed benchmark can measure fact linking performance on a granular triple slot level while also measuring if a system can recognize that a surface form has no match in the existing KG. |
AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents (2024.naacl-demo)
Copied to clipboard
Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, Carolin Lawrence
| Challenge: | Existing benchmarks are narrow and simply compute overall task success. |
| Approach: | They propose a framework where both benchmarks and metrics are modular and easily extensible through well documented and easy-to-use APIs. |
| Outcome: | The proposed framework can track agent progress on two use cases and identify common failure points and refine the agent architecture to obtain a significant performance increase. |
Generating and Evaluating Plausible Explanations for Knowledge Graph Completion (2024.acl-long)
Copied to clipboard
| Challenge: | Existing XAI approaches focus on learning algorithmic explanations, but are not plausible for users. |
| Approach: | They propose a path-based explanation method that meets human-centric explainability constraints and enhances plausibility. |
| Outcome: | The proposed method meets human-centric explainability constraints and enhances plausibility. |
On Synthesizing Data for Context Attribution in Question Answering (2025.acl-long)
Copied to clipboard
Gorjan Radevski, Kiril Gashteovski, Shahbaz Syed, Christopher Malon, Sebastien Nicolas, Chia-Chien Hung, Timo Sztyler, Verena Heußer, Wiem Ben Rim, Masafumi Enomoto, Kunihiro Takeoka, Masafumi Oyamada, Goran Glavaš, Carolin Lawrence
| Challenge: | Large Language Models (LLMs) have a tendency to hallucinate, resulting in false or misleading answers. |
| Approach: | They propose a novel generative strategy for synthesizing context attribution data. |
| Outcome: | The proposed approach is highly effective for fine-tuning small LMs for context attribution in different QA tasks and domains. |
MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis (2025.acl-long)
Copied to clipboard
Daniel Philip Rose, Chia-Chien Hung, Marco Lepri, Israa Alqassem, Kiril Gashteovski, Carolin Lawrence
| Challenge: | Existing approaches to DDx are limited by single-dataset evaluations, isolated optimization of components, unrealistic assumptions about complete patient profiles, and single-attempt diagnosis. |
| Approach: | They propose a Modular Explainable DDx Agent framework that allows physicians to iteratively refine a ranked list of possible diseases based on symptoms, antecedents, and medical knowledge. |
| Outcome: | The proposed framework achieves over 10% accuracy improvements in interactive DDx across large and small LLMs while offering critical explainability into its diagnostic reasoning process. |
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback (P18-1)
Copied to clipboard
| Challenge: | a recent study shows that counterfactual learning from human bandit feedback can improve neural semantic parsers . cost and difficulty of manually preparing large amounts of parses is a bottleneck for supervised learning . |
| Approach: | They propose to use human bandit feedback to apply counterfactual learning to neural parsing . they devise an easy-to-use interface to collect human feedback on semantic parses . |
| Outcome: | The proposed framework improves semantic parsers by reducing the cost of manual parsing . the proposed framework is based on human bandit feedback collected by the user . |
A Human-Centric Evaluation Platform for Explainable Knowledge Graph Completion (2024.eacl-demo)
Copied to clipboard
| Challenge: | Evaluating plausibility, the helpfulness of explanations, is essential for developing eXplainable AI (XAI) that can really aid human users. |
| Approach: | They propose a human-centric evaluation platform to measure plausibility of explanations in the context of eXplainable Knowledge Graph Completion (XKGC) they showcase two use cases to illustrate what results can be achieved with the system. |
| Outcome: | The proposed evaluation platform is designed to evaluate plausibility of explanations in eXplainable Knowledge Graph Completion (XKGC) the proposed evaluation system is based on two use cases in an experimental setting to demonstrate the results. |
Compositional Steering of Large Language Models with Steering Tokens (2026.acl-long)
Copied to clipboard
| Challenge: | Existing work addresses compositional steering for a single behavior, but it's limited by computational costs and performance. |
| Approach: | They propose compositional steering tokens for multi-behavior steering by embedding individual behaviors into dedicated tokens via self-distillation. |
| Outcome: | The proposed steering tokens generalize well to unseen compositions compared to competing approaches. |
Attending to Future Tokens for Bidirectional Sequence Generation (D19-1)
Copied to clipboard
| Challenge: | Neural sequence generation is typically performed token-by-token and left-to-right. |
| Approach: | They propose to use placeholder tokens to make the sequence generation process bidirectional by taking past and future tokens into consideration when generating the actual output token. |
| Outcome: | The proposed approach outperforms baselines on two conversational tasks by a large margin. |
MILIE: Modular & Iterative Multilingual Open Information Extraction (2022.acl-long)
Copied to clipboard
Bhushan Kotnis, Kiril Gashteovski, Daniel Rubio, Ammar Shaker, Vanesa Rodriguez-Tembras, Makoto Takamoto, Mathias Niepert, Carolin Lawrence
| Challenge: | Current OpenIE systems extract all triple slots independently. |
| Approach: | They propose a neural OpenIE system that extracts triple slots iteratively . they propose to use the system to extract easy slots and difficult ones . |
| Outcome: | The proposed system outperforms SOTA systems on multiple languages ranging from Chinese to Arabic. |
KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models (2022.emnlp-demos)
Copied to clipboard
Haris Widjaja, Kiril Gashteovski, Wiem Ben Rim, Pengfei Liu, Christopher Malon, Daniel Ruffinelli, Carolin Lawrence, Graham Neubig
| Challenge: | Knowledge Graphs (KGs) store information in the form of (head, predicate, tail)-triples. |
| Approach: | They propose a framework for performing fine-grained evaluation on meaningful subsets of data. |
| Outcome: | The proposed framework tests models on meaningful subsets of the data, which would have been impossible to detect with standard averaged single-score metrics. |
AnnIE: An Annotation Platform for Constructing Complete Open Information Extraction Benchmark (2022.acl-demo)
Copied to clipboard
Niklas Friedrich, Kiril Gashteovski, Mingying Yu, Bhushan Kotnis, Carolin Lawrence, Mathias Niepert, Goran Glavaš
| Challenge: | Open Information Extraction (OIE) is the task of extracting facts from sentences in the form of relations and their corresponding arguments in schema-free manner. |
| Approach: | They propose an interactive annotation platform that facilitates annotating complete facts from input sentences. |
| Outcome: | The proposed platform facilitates such challenging annotation tasks and supports creation of fact-oriented OIE evaluation benchmarks. |