Papers by Kiril Gashteovski
TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Action (2026.findings-eacl)
Copied to clipboard
| Challenge: | Humanitarian Mine Action (HMA) authorities publish large amount of life-saving operational knowledge, but much remains locked away in unstructured reports. |
| Approach: | They propose a dataset, evaluation framework and ontology-guided large language model pipeline for knowledge extraction from text in the HMA domain. |
| Outcome: | The proposed framework improves extraction accuracy by 44.2% and reduces hallucinations by 22.5% . the proposed framework can be used to analyze human-annotated triples and an LLM-as-Judge protocol . |
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions (2025.emnlp-main)
Copied to clipboard
Emmy Liu, Amanda Bertsch, Lintang Sutawika, Lindia Tjuatja, Patrick Fernandes, Lara Marinov, Michael Chen, Shreya Singhal, Carolin Lawrence, Aditi Raghunathan, Kiril Gashteovski, Graham Neubig
| Challenge: | Language model performance is largely dependent on pretraining decisions, but scaling laws based on only these two aspects do not always explain downstream task performance. |
| Approach: | They meta-analyze 92 open-source pretrained models to quantify their impact on performance. |
| Outcome: | The framework lays a foundation for more systematic investigation of how model development choices shape final capabilities. |
Can We Predict New Facts with Open Knowledge Graph Embeddings? A Benchmark for Open Link Prediction (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for predicting knowledge graphs rely on the rich structure of the knowledge graph. |
| Approach: | They propose an evaluation protocol and a methodology for creating the open link prediction benchmark OlpBench. |
| Outcome: | The proposed model predicts test facts by completing questions in open link prediction task. |
Evaluating Language Models as Synthetic Data Generators (2025.acl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, Graham Neubig
| Challenge: | Prior studies have focused on developing effective data generation methods, but lack systematic comparison of different LMs as data generators in a unified setting. |
| Approach: | They propose to use a benchmark to compare language models' data generation abilities against a set of standardized settings and metrics. |
| Outcome: | The proposed benchmark provides standardized settings and metrics to evaluate LMs’ data generation abilities. |
Scaling Evaluation-Time Compute with Reasoning Models as Evaluators (2026.findings-acl)
Copied to clipboard
Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Minkyeong Moon, Carolin Lawrence, Kiril Gashteovski, Julia Hockenmaier, Graham Neubig, Sean Welleck
| Challenge: | Language model (LM) evaluators that generate chain-of-thought reasoning are widely used for the assessment of LM responses. |
| Approach: | They investigate whether increasing LMs' "thinking" time through scaling test-time compute can improve an LM's evaluation capability. |
| Outcome: | The proposed reasoning models improve evaluation performance monotonically with the number of reasoning tokens generated, mirroring trends seen in LM reasoning. |
Robust Text Classification: Analyzing Prototype-Based Networks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language models exhibit a drop in performance on noisy data, which can cause classifiers to incorrectly change their predictions. |
| Approach: | They propose to use Prototype-Based Networks to classify examples based on their similarity to prototypical examples of a class (prototypes) they show that PBNs offer more robustness under both targeted and static adversarial attacks. |
| Outcome: | The proposed model is robust to noise and targets both targeted and static attacks. |
BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for OIE are incomplete and do not include all acceptable variants of the same fact. |
| Approach: | They introduce BenchIE: a benchmark and evaluation framework for comprehensive evaluation of OIE systems for English, Chinese, and German. |
| Outcome: | The proposed framework is based on fact synsets, clusters, and standardized benchmarks. |
Linking Surface Facts to Large-Scale Knowledge Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Open Information Extraction (OIE) methods extract facts in the form of triples . ambiguity of these triples hinders their downstream usage . |
| Approach: | They propose a benchmark that measures fact linking performance on a granular triple slot level . they propose to use a system that can detect out-of-KG entities and predicates . |
| Outcome: | The proposed benchmark can measure fact linking performance on a granular triple slot level while also measuring if a system can recognize that a surface form has no match in the existing KG. |
AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents (2024.naacl-demo)
Copied to clipboard
Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, Carolin Lawrence
| Challenge: | Existing benchmarks are narrow and simply compute overall task success. |
| Approach: | They propose a framework where both benchmarks and metrics are modular and easily extensible through well documented and easy-to-use APIs. |
| Outcome: | The proposed framework can track agent progress on two use cases and identify common failure points and refine the agent architecture to obtain a significant performance increase. |
On Synthesizing Data for Context Attribution in Question Answering (2025.acl-long)
Copied to clipboard
Gorjan Radevski, Kiril Gashteovski, Shahbaz Syed, Christopher Malon, Sebastien Nicolas, Chia-Chien Hung, Timo Sztyler, Verena Heußer, Wiem Ben Rim, Masafumi Enomoto, Kunihiro Takeoka, Masafumi Oyamada, Goran Glavaš, Carolin Lawrence
| Challenge: | Large Language Models (LLMs) have a tendency to hallucinate, resulting in false or misleading answers. |
| Approach: | They propose a novel generative strategy for synthesizing context attribution data. |
| Outcome: | The proposed approach is highly effective for fine-tuning small LMs for context attribution in different QA tasks and domains. |
MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis (2025.acl-long)
Copied to clipboard
Daniel Philip Rose, Chia-Chien Hung, Marco Lepri, Israa Alqassem, Kiril Gashteovski, Carolin Lawrence
| Challenge: | Existing approaches to DDx are limited by single-dataset evaluations, isolated optimization of components, unrealistic assumptions about complete patient profiles, and single-attempt diagnosis. |
| Approach: | They propose a Modular Explainable DDx Agent framework that allows physicians to iteratively refine a ranked list of possible diseases based on symptoms, antecedents, and medical knowledge. |
| Outcome: | The proposed framework achieves over 10% accuracy improvements in interactive DDx across large and small LLMs while offering critical explainability into its diagnostic reasoning process. |
A Human-Centric Evaluation Platform for Explainable Knowledge Graph Completion (2024.eacl-demo)
Copied to clipboard
| Challenge: | Evaluating plausibility, the helpfulness of explanations, is essential for developing eXplainable AI (XAI) that can really aid human users. |
| Approach: | They propose a human-centric evaluation platform to measure plausibility of explanations in the context of eXplainable Knowledge Graph Completion (XKGC) they showcase two use cases to illustrate what results can be achieved with the system. |
| Outcome: | The proposed evaluation platform is designed to evaluate plausibility of explanations in eXplainable Knowledge Graph Completion (XKGC) the proposed evaluation system is based on two use cases in an experimental setting to demonstrate the results. |
Compositional Steering of Large Language Models with Steering Tokens (2026.acl-long)
Copied to clipboard
| Challenge: | Existing work addresses compositional steering for a single behavior, but it's limited by computational costs and performance. |
| Approach: | They propose compositional steering tokens for multi-behavior steering by embedding individual behaviors into dedicated tokens via self-distillation. |
| Outcome: | The proposed steering tokens generalize well to unseen compositions compared to competing approaches. |
MILIE: Modular & Iterative Multilingual Open Information Extraction (2022.acl-long)
Copied to clipboard
Bhushan Kotnis, Kiril Gashteovski, Daniel Rubio, Ammar Shaker, Vanesa Rodriguez-Tembras, Makoto Takamoto, Mathias Niepert, Carolin Lawrence
| Challenge: | Current OpenIE systems extract all triple slots independently. |
| Approach: | They propose a neural OpenIE system that extracts triple slots iteratively . they propose to use the system to extract easy slots and difficult ones . |
| Outcome: | The proposed system outperforms SOTA systems on multiple languages ranging from Chinese to Arabic. |
KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models (2022.emnlp-demos)
Copied to clipboard
Haris Widjaja, Kiril Gashteovski, Wiem Ben Rim, Pengfei Liu, Christopher Malon, Daniel Ruffinelli, Carolin Lawrence, Graham Neubig
| Challenge: | Knowledge Graphs (KGs) store information in the form of (head, predicate, tail)-triples. |
| Approach: | They propose a framework for performing fine-grained evaluation on meaningful subsets of data. |
| Outcome: | The proposed framework tests models on meaningful subsets of the data, which would have been impossible to detect with standard averaged single-score metrics. |
AnnIE: An Annotation Platform for Constructing Complete Open Information Extraction Benchmark (2022.acl-demo)
Copied to clipboard
Niklas Friedrich, Kiril Gashteovski, Mingying Yu, Bhushan Kotnis, Carolin Lawrence, Mathias Niepert, Goran Glavaš
| Challenge: | Open Information Extraction (OIE) is the task of extracting facts from sentences in the form of relations and their corresponding arguments in schema-free manner. |
| Approach: | They propose an interactive annotation platform that facilitates annotating complete facts from input sentences. |
| Outcome: | The proposed platform facilitates such challenging annotation tasks and supports creation of fact-oriented OIE evaluation benchmarks. |
Leveraging Open Information Extraction for More Robust Domain Transfer of Event Trigger Detection (2024.findings-eacl)
Copied to clipboard
| Challenge: | Event detection is a crucial information extraction task in many domains, such as Wikipedia or news. |
| Approach: | They propose to use subject-object relations obtained from a rule-based open information extraction system to enhance domain transfer of trigger detection (TD) they combine this enhanced transfer with masked language modeling on the target domain, observing further TD transfer gains. |
| Outcome: | The proposed model improves the transfer of triggers between domains and reduces performance drops when using a low-resource source domain to a high-res target domain. |