| Challenge: | KnowledgeNet provides text exhaustively annotated with facts . high-quality KBs still rely almost exclusively on human-curated structured or semi-structured data. |
| Approach: | They propose five baseline approaches to populating a knowledge base with facts . the best approach achieves an F1 score of 0.50, significantly outperforming a traditional approach by 79% . |
| Outcome: | The best approach achieves an F1 score of 0.50, outperforming a traditional approach by 79%, indicating the dataset is challenging. |
Similar Papers
Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations on the population task are either not accurate (automatic evaluation with randomly sampled negative examples) or of small scale (human annotation). |
| Approach: | They propose a reasoning over commonsense knowledge bases (CSKBs) that are free-text and have a human annotation set to probe commonsensical reasoning. |
| Outcome: | The proposed model is based on a human-annotated evaluation set and is compared with existing models on the population task. |
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers (2024.findings-emnlp)
Copied to clipboard
Yuxia Wang, Revanth Gangi Reddy, Zain Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models generate naturally sounding answers over a broad range of human inquiries, but they often generate answers that contradict real-world facts. |
| Approach: | They propose a framework for annotating and evaluating the factuality of large language models . they propose 'factcheck-bench' which provides a multi-stage annotation scheme . |
| Outcome: | The proposed framework outperforms several popular LLM fact-checkers in claim, sentence, and document levels. |
Evaluation Dataset and Methodology for Extracting Application-Specific Taxonomies from the Wikipedia Knowledge Graph (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent efforts to extract hierarchical relations from unstructured text have been challenging. |
| Approach: | They propose an iterative method to extract an application-specific gold standard dataset from a Wikipedia knowledge graph and an evaluation framework to assess the quality of noisy automatically extracted taxonomies. |
| Outcome: | The proposed method reduces manual work and provides a first gold standard dataset and evaluation framework. |
A Short Survey on Sense-Annotated Corpora (2020.lrec-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a key task in Natural Language Understanding. |
| Approach: | They propose to use sense-annotated corpora for supervised Word Sense Disambiguation. |
| Outcome: | The proposed methods have been compared with knowledge-based approaches and have shown to be more efficient when they are available. |
WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation (2021.acl-short)
Copied to clipboard
| Challenge: | Existing summarization datasets are limited in their ability to evaluate output . a human evaluation is necessary to understand and improve summarizing systems . |
| Approach: | They propose a dataset based on how-to articles and coherent paragraph summaries written in plain language. |
| Outcome: | The proposed dataset makes human evaluation easier and more effective . the authors compare the proposed dataset to existing ones on PubMed and the literature. |
WordNet under Scrutiny: Dictionary Examples in the Era of Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Lexical resources are a repository of knowledge and are used for many tasks, including word sense disambiguation and etymology. |
| Approach: | They compare WordNet, the most commonly used lexical resource in NLP, with a variety of dictionaries and examples that were generated by ChatGPT. |
| Outcome: | The most commonly used lexical resource in NLP, with a variety of dictionaries and examples that were generated by ChatGPT. |
Inspecting the concept knowledge graph encoded by modern language models (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are used to solve tasks such as summarization and information retrieval. |
| Approach: | They propose to use word embeddings, text generators, context encoders to extract underlying knowledge graphs of nine influential language models. |
| Outcome: | The proposed model is able to encode word embeddings, text generators, and context encoders, but suffers from several inaccuracies. |
A guide to the dataset explosion in QA, NLI, and commonsense reasoning (2020.coling-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to provide an up-to-date guide to the recent datasets . the target audience is the NLP practitioners who are lost in dozens of the recent data sets. |
| Approach: | This tutorial provides an up-to-date guide to the recent datasets . it surveys old and new methodological issues with dataset construction . |
| Outcome: | This tutorial aims to provide an up-to-date guide to the recent datasets . it surveys the old and new methodological issues with dataset construction . |
GenWiki: A Dataset of 1.3 Million Content-Sharing Text and Graphs for Unsupervised Graph-to-Text Generation (2020.coling-main)
Copied to clipboard
| Challenge: | a large-scale, general-domain dataset is needed for knowledge graph-to-text generation . data collection is expensive and data-intensive, making it difficult to get good annotation . |
| Approach: | They propose to use a large-scale, general-domain dataset to generate unsupervised text from knowledge graphs. |
| Outcome: | The proposed dataset has 1.3M text and graph examples, and is a benchmark for future research . good annotation is expensive and difficult to get, and it's difficult to check quality . |
A Decade of Knowledge Graphs in Natural Language Processing: A Survey (2022.aacl-main)
Copied to clipboard
| Challenge: | Knowledge graphs (KGs) are a representation of semantic relations between entities . despite their popularity, there is still no general understanding of what exactly a KG is or for what tasks it is applicable. |
| Approach: | They analyze 507 papers on knowledge graphs in natural language processing (NLP) they provide a taxonomy of tasks and review the maturity of individual research streams . |
| Outcome: | The findings summarize the literature and highlight directions for future work. |