When ACE met KBP: End-to-End Evaluation of Knowledge Base Population with Component-level Annotation (L18-1)
Copied to clipboard
| Challenge: | Automating constructing a Knowledge Base from unstructured text is a goal of natural language processing. |
| Approach: | They propose a method to evaluate a Knowledge Base population from unstructured text . they propose bootstrap resampling to provide statistical significance to the results . |
| Outcome: | The proposed method uses component-level annotations to evaluate Cold Start KBP . it also uses bootstrap resampling to provide statistical significance to the results reported . |
Similar Papers
Laying the Groundwork for Knowledge Base Population: Nine Years of Linguistic Resources for TAC KBP (L18-1)
Copied to clipboard
| Challenge: | Knowledge Base Population (KBP) evaluations target information extraction technologies for knowledge bases comprised of entities, relations, and events. |
| Approach: | They describe the linguistic resources provided by Linguistic Data Consortium for TAC KBP since 2009 . they highlight changes made to support evolving evaluation requirements . |
| Outcome: | The evaluations have targeted information extraction technologies for the population of knowledge bases comprised of entities, relations, and events. |
FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools to evaluate long text outputs are lacking in the field of NLP . human rating and error analysis remains a crucial component for any evaluation of long text generation. |
| Approach: | They propose a web-based toolkit to collect fine-grained error annotations for long texts . they use a taxonomy to identify errors and assign them to text spans . |
| Outcome: | The proposed tool can be used to evaluate the coherence of long generated summaries. |
Set Generation Networks for End-to-End Knowledge Base Population (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge base population systems require a machine translation task to generate multiple facts, but the fact order is not considered. |
| Approach: | They propose a knowledge base population task that aims to discover facts about entities from texts and expand a KB with these facts. |
| Outcome: | The proposed networks achieve state-of-the-art (SoTA) performance on two benchmark datasets. |
Let’s Stop Incorrect Comparisons in End-to-end Relation Extraction! (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature on Relation Extraction (RE) uses multiple evaluation setups to compare performance. |
| Approach: | They propose to quantify the most common comparison mistake and evaluate it leads to overestimating the final RE performance by around 5% on ACE05. |
| Outcome: | The proposed meta-analysis overestimates the final RE performance by around 5% on ACE05. |
Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks, but statistical bias in benchmark data and probing studies has recently called into question their true capabilities. |
| Approach: | They propose to evaluate systems through a measure of prediction coherence by using two existing language understanding benchmarks with different properties to demonstrate its versatility. |
| Outcome: | The proposed evaluation framework is quick, effective, and versatile to provide insight into the coherence of machines’ predictions. |
Evaluating Zero-Shot Event Structures: Recommendations for Automatic Content Extraction (ACE) Annotations (2023.acl-short)
Copied to clipboard
| Challenge: | Zero-shot event extraction (EE) methods infer richly structured event records from unstructured text data, based on a user-supplied natural language specification and no training examples. |
| Approach: | They propose recommendations for future evaluations so the research community can better utilize ACE as an event evaluation resource. |
| Outcome: | The proposed methods can be used to evaluate zero-shot and other low-supervision EE methods, considering up to 32% of correctly identified arguments and 25% of correctly ignored event mentions as false negatives. |
End-to-End Construction of NLP Knowledge Graph (2021.findings-acl)
Copied to clipboard
| Challenge: | a new schema for NLP knowledge about tasks, datasets and metrics is proposed. |
| Approach: | They propose a new schema that represents knowledge about tasks, datasets and metrics in the NLP domain. |
| Outcome: | The proposed framework can be automatically built into scientific leaderboards . the proposed system achieves reasonable results for all relation types on this small-scale graph . |
Evaluating the Knowledge Base Completion Potential of GPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC) however, their ability to do this at scale and with high accuracy remains an open question. |
| Approach: | They propose to use language models to complete a large public KB, Wikidata, with 90% precision. |
| Outcome: | The proposed models can extend Wikidata by 27M facts at 90% precision. |
LightTag: Text Annotation Platform (2021.emnlp-demo)
Copied to clipboard
| Challenge: | LightTag is a text annotation tool built on the premise of global optimization by addressing annotator as well as project managers and data scientists who manage the work and enforce production quality. |
| Approach: | They propose to use LightTag to optimize the global NLP process by addressing annotators as well as project managers and data scientists who manage the work and enforce production quality. |
| Outcome: | The proposed tool is based on the theory of constraints and is available for free for academic use. |
A Short Survey on Sense-Annotated Corpora (2020.lrec-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a key task in Natural Language Understanding. |
| Approach: | They propose to use sense-annotated corpora for supervised Word Sense Disambiguation. |
| Outcome: | The proposed methods have been compared with knowledge-based approaches and have shown to be more efficient when they are available. |