| Challenge: | Evaluating annotator consistency is crucial when building datasets for mention detection. |
| Approach: | They propose to use different fuzzy-matching functions to resolve this ambiguity by extracting syntactic heads present in annotations and using the Dice coefficient to measure similarity between sets. |
| Outcome: | The proposed functions are tested against the judgment of a human evaluator and show that the best-performing function agrees with the human . |
Similar Papers
Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators (2026.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art NLP models are expensive and inefficient for event annotation. |
| Approach: | They propose to integrate LLMs into a holistic workflow that summarizes news with event coreference resolution and argument extraction in three modes: AI-only, AI assistance, and human only. |
| Outcome: | The proposed workflow integrates LLMs to alleviate human labor in a holistic pipeline. |
TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction (2024.findings-acl)
Copied to clipboard
Kuan-Hao Huang, I-Hung Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Prem Natarajan, Kai-Wei Chang, Nanyun Peng, Heng Ji
| Challenge: | Recent studies suggest that event extraction evaluations may not accurately reflect the true performance. |
| Approach: | They propose a standardized, fair, and reproducible benchmark for event extraction . they use standardized scripts and splits for 16 datasets spanning eight domains . |
| Outcome: | The proposed benchmarks show that they struggle to achieve satisfactory performance. |
DEIE: Benchmarking Document-level Event Information Extraction with a Large-scale Chinese News Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing event-based datasets mainly target sentence-level tasks . current models struggle with "document" annotation, a key feature of the current model . |
| Approach: | They propose a large-scale document-level event information extraction dataset with over 56,000+ events and 242,000+ arguments. |
| Outcome: | The proposed dataset has over 56,000+ events and 242,000+ arguments. |
Fusion meets Function: The Adaptive Selection-Generation Approach in Event Argument Extraction (2025.coling-main)
Copied to clipboard
| Challenge: | Event Argument Extraction is a critical subtask of Event Extraction, focused on identifying event arguments within text. |
| Approach: | They propose a Fusion Selection-Generation-Based Approach that merges selective and generative methods to enhance argument extraction accuracy. |
| Outcome: | The proposed method improves on the RAMS and WikiEvents, while preserving the unique characteristics of both methods. |
Event Linking: Grounding Event Mentions to Wikipedia (2023.eacl-main)
Copied to clipboard
| Challenge: | a new task for natural language understanding is called Event Linking . the context where an event is mentioned lacks the details of this event . |
| Approach: | They propose a new task to link an article's event mention to the most appropriate Wikipedia page . they collect a training set from Wikipedia and evaluate two models to test the task . |
| Outcome: | The proposed model is based on a dataset and a real-world news domain . it is expected that the most appropriate Wikipedia page will provide rich knowledge about the mention . |
EventRelBench: A Comprehensive Benchmark for Evaluating Event Relation Understanding in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing LLMs fail to capture event relationships, despite advances in NLP . a new benchmark is being developed to assess LLM's ability to extract event relationships . |
| Approach: | They propose a benchmark to assess LLMs' ability to extract event relations . EventRelBench comprises 35K diverse event relation questions . |
| Outcome: | The benchmark EventRelBench measures the performance of large language models on event relation extraction tasks. |
Event-Centric Natural Language Processing (2021.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial will provide an introduction to various methods for automating the extraction, conceptualization and prediction of events and their relations. |
| Approach: | This tutorial will provide an introduction to various methods for automating events and their relations, and a wide range of NLU and commonsense understanding tasks. |
| Outcome: | This tutorial will provide an introduction to various methods for automating extraction, conceptualization and prediction of events and their relations, and a wide range of NLU and commonsense understanding tasks. |
REGen: A Reliable Evaluation Framework for Generative Event Argument Extraction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work evaluates event argument extraction with exact match (EM), where predicted arguments must align exactly with annotated spans. |
| Approach: | They propose a Reliable Evaluation framework for Generative event argument extraction that combines exact, relaxed, and LLM-based matching to better align with human judgment. |
| Outcome: | Experiments on six datasets show that REGen achieves an average performance gain of +23.93 F1 over EM, reflecting capabilities overlooked by prior evaluation. |
Conundrums in Event Coreference Resolution: Making Sense of the State of the Art (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent years have seen the successful application of span-based neural models to entity-based information extraction tasks such as entity coreference resolution (CR) Existing event coreference resolvers focused on feature engineering are few and far between, let alone event corefers. |
| Approach: | They propose to adapt existing span-based event reference systems to event coreference by adapting the models originally developed for entity coreference to event CR. |
| Outcome: | The proposed model improves the representations of entity mentions in entity-based IE tasks compared to non-span models . |
Don’t Annotate, but Validate: a Data-to-Text Method for Capturing Event Data (L18-1)
Copied to clipboard
| Challenge: | Existing methods to create event data are limited by ambiguity and variation in the data. |
| Approach: | They propose a method to obtain large volumes of text corpora with event data . they use a tool to annotate texts and enrich the reference texts with event coreference annotations. |
| Outcome: | The proposed method obtains large volumes of high-quality text corpora with event data . the data obtained with this method have high precision and at a large scale . |