SciNLP: A Domain-Specific Benchmark for Full-Text Scientific Entity and Relation Extraction in NLP (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets for structured information extraction focus on specific publication sections due to domain complexity and high cost of annotating scientific texts. |
| Approach: | They propose a specialized benchmark for full-text entity and relation extraction in the natural language processing domain. |
| Outcome: | The proposed dataset comprises 60 manually annotated full-text NLP publications covering 7,072 entities and 1,826 relations. |
Similar Papers
SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Scientific information extraction (SciIE) is critical for converting unstructured knowledge from scholarly articles into structured data. |
| Approach: | They propose to use a scientific entity and relation extraction dataset to capture interactions between entities in full texts. |
| Outcome: | The proposed dataset captures the intricate use and interactions among entities in full texts and provides an out-of-distribution test set to offer a more realistic evaluation. |
SciNLI: A Corpus for Natural Language Inference on Scientific Text (2022.acl-long)
Copied to clipboard
| Challenge: | Existing Natural Language Inference (NLI) datasets are not related to scientific text. |
| Approach: | They propose a large dataset for NLI that captures the formality in scientific text and contains 107,412 sentence pairs extracted from scholarly papers on NLP and computational linguistics. |
| Outcome: | The proposed model achieves a Macro F1 score of only 78.18% and an accuracy of 78.23%. |
End-to-End Construction of NLP Knowledge Graph (2021.findings-acl)
Copied to clipboard
| Challenge: | a new schema for NLP knowledge about tasks, datasets and metrics is proposed. |
| Approach: | They propose a new schema that represents knowledge about tasks, datasets and metrics in the NLP domain. |
| Outcome: | The proposed framework can be automatically built into scientific leaderboards . the proposed system achieves reasonable results for all relation types on this small-scale graph . |
Pushing the Frontiers of Scientific Fact-Checking: The SCINLP Dataset (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly being used to understand how scientific research evolves, drawing growing interest from the research community. |
| Approach: | They propose a scientific fact-checking dataset, SCINLP, tailored to the NLP domain that verifies the veracity of scientific research questions across varying rationale contexts. |
| Outcome: | The proposed framework examines scientific claims and research focus from a curated collection of influential and reputable NLP papers published between 2000 and 2024. |
SciEvent: Benchmarking Multi-domain Scientific Event Extraction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on scientific information extraction relies on entity-relation extraction in narrow domains . current models struggle in domains such as sociology and humanities . |
| Approach: | They propose a multi-domain benchmark for scientific abstract annotations using a unified event extraction schema. |
| Outcome: | The proposed benchmark includes 500 abstracts across five research domains with manual annotations of event segments, triggers, and fine-grained arguments. |
SciDMT: A Large-Scale Corpus for Detecting Scientific Mentions (2024.lrec-main)
Copied to clipboard
| Challenge: | SciDMT is an enhanced and expanded corpus for scientific mention detection . existing corpora are limited by their small volume and entity linking capabilities . |
| Approach: | They propose to enhance SciDMT, an annotated scientific corpus for scientific mention detection. |
| Outcome: | The proposed corpus is the largest for scientific entity mention detection . it is based on deep learning architectures like SciBERT and GPT-3.5 . |
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on sentence-level inference, which limits its application in downstream NLP problems. |
| Approach: | They propose to construct a large-scale dataset for document-level NLI that can be used to study NLP problems. |
| Outcome: | The proposed model performs well on popular sentence-level benchmarks and generalizes well to out-of-domain NLP tasks that rely on inference at document granularity. |
SciDTB: Discourse Dependency TreeBank for Scientific Abstracts (P18-2)
Copied to clipboard
| Challenge: | Discourse relations are annotated on scientific articles. |
| Approach: | They propose a domain-specific discourse treebank annotated on scientific articles . they use dependency trees to represent discourse structure, which is flexible and simplified . |
| Outcome: | The proposed treebank is a benchmark for evaluating discourse dependency parsers. |
MedDistant19: Towards an Accurate Benchmark for Broad-Coverage Biomedical Relation Extraction (2022.coling-1)
Copied to clipboard
| Challenge: | Relation extraction in the biomedical domain is challenging due to the lack of labeled data and high annotation costs. |
| Approach: | They propose to use distant supervision to pair knowledge graph relationships with raw texts to tackle the scarcity of annotated data and to validate their results. |
| Outcome: | The proposed benchmarks are more accurate and consistent with existing benchmarks and show that there is no train-test leakage. |
CogCompNLP: Your Swiss Army Knife for NLP (L18-1)
Copied to clipboard
Daniel Khashabi, Mark Sammons, Ben Zhou, Tom Redman, Christos Christodoulopoulos, Vivek Srikumar, Nicholas Rizzolo, Lev Ratinov, Guanheng Luo, Quang Do, Chen-Tse Tsai, Subhro Roy, Stephen Mayhew, Zhili Feng, John Wieting, Xiaodong Yu, Yangqiu Song, Shashank Gupta, Shyam Upadhyay, Naveen Arivazhagan, Qiang Ning, Shaoshi Ling, Dan Roth
| Challenge: | a corpus-reader module supports popular corpora, feature extraction and annotation modules for semantic and syntactic tasks. |
| Approach: | They propose a library that provides modules to address different challenges . they provide a corpus-reader module that supports popular corpora in the NLP community . |
| Outcome: | The proposed library simplifies the process of design and development of NLP applications by providing modules to address different challenges. |