Towards Robust Universal Information Extraction: Dataset, Evaluation, and Solution (2025.acl-long)
Copied to clipboard
| Challenge: | Existing robust benchmark datasets generate only a limited range of perturbations for a single Information Extraction (UIE) task, which fails to evaluate the robustness of UIE models effectively. |
| Approach: | They propose a new benchmark dataset that utilizes Large Language Models to generate more diverse and realistic perturbations across different IE tasks. |
| Outcome: | The proposed model performs better with only 15% of the data and is more robust with other models. |
Similar Papers
RUIE: Retrieval-based Unified Information Extraction using Large Language Model (2025.coling-main)
Copied to clipboard
| Challenge: | Unified information extraction (UIE) aims to extract diverse structured information from unstructured text using a single model or framework. |
| Approach: | They propose a framework that leverages in-context learning for efficient task generalization by combining LLM preferences with a keyword-enhanced reward model. |
| Outcome: | The proposed framework performs better on eight held-out datasets than existing methods and instruction-tuning methods. |
Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks focus on pairwise matching, ignoring robustness . current models exhibit frustrating degradation, with a maximum drop of 23.43 F1 score . |
| Approach: | They propose a benchmark that simulates the evaluation of open information extraction models in the real world . they perform experiments on typical models published in the last decade and a representative large language model . |
| Outcome: | The proposed model is rated robust on a knowledge-invariant clique with different syntactic and expressive forms. |
ProUIE: A Macro-to-Micro Progressive Learning Method for LLM-based Universal Information Extraction (2026.findings-acl)
Copied to clipboard
Wenda Liu, Song Zhigang, Shuai Nie, Guangyao Liu, Lisung Chen, Binyu Yang, Yaran Chen, Peng Zhou, Hongzhen Wang, Yuchen Liu, Wenyue Hu, Jiaming Xu, Runyu Shi, Ying Huang
| Challenge: | ProUIE improves universal information extraction (UIE) without external information . many LLM-based methods rely on extra schema cues, external resources or complex alignment and verification pipelines . |
| Approach: | They propose a Macro-to-Micro progressive learning approach that improves UIE without external information. |
| Outcome: | ProUIE outperforms instruction-tuned baselines on average for NER and RE while using a smaller backbone. |
Adversarial NLI: A New Benchmark for Natural Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | a new large-scale NLI benchmark dataset is presented to test models on a variety of popular NLIs. |
| Approach: | They propose a large-scale NLI benchmark dataset that is iteratively compared with a human-and-model-in-the-loop procedure. |
| Outcome: | The proposed method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate. |
MMUIE: Massive Multi-Domain Universal Information Extraction for Long Documents (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing document-level information extraction systems operate at the sentence level or within narrow domains due to annotation constraints. |
| Approach: | They propose a large-scale universal dataset for multi-domain, document-level information extraction from long texts. |
| Outcome: | The proposed dataset integrates traditional knowledge bases with large language models to extract fine-grained entities, aliases, and relation triples across 34 domains. |
BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for OIE are incomplete and do not include all acceptable variants of the same fact. |
| Approach: | They introduce BenchIE: a benchmark and evaluation framework for comprehensive evaluation of OIE systems for English, Chinese, and German. |
| Outcome: | The proposed framework is based on fact synsets, clusters, and standardized benchmarks. |
Improving Open Information Extraction via Iterative Rank-Aware Learning (P19-1)
Copied to clipboard
| Challenge: | Open information extraction (IE) is the task of extracting open-domain assertions from natural language sentences. |
| Approach: | They propose an additional binary classification loss to calibrate the extraction likelihood . they propose an iterative learning process where extractions generated by the open IE model are incrementally included as training samples to help the model learn from trial and error. |
| Outcome: | Experiments on open information extraction (IE) show that the extraction likelihood is not well calibrated when comparing quality of extracted assertions. |
Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve robustness require changing the fine-tuning process or large-scale data augmentation, which are infeasible or cost prohibitive for closed-source models. |
| Approach: | They propose to prioritize more complex examples or replace existing training examples with LLM-generated data to improve performance on OOD NLI datasets. |
| Outcome: | The proposed methods improve performance on difficult OOD datasets while training with synthetic data leads to substantial improvements on easier OOD data. |
TRUE-UIE: Two Universal Relations Unify Information Extraction Tasks (2024.naacl-long)
Copied to clipboard
| Challenge: | Information extraction (IE) tasks have a variety of schemas and objectives that differ across tasks. |
| Approach: | They propose a paradigm where all IE tasks are aligned to learn the same goals . they use two universal relations to extract mention spans and type recognition . |
| Outcome: | The proposed model achieves state-of-the-art on established benchmarks spanning 16 datasets, spanning 7 diverse IE tasks. |
LSOIE: A Large-Scale Dataset for Supervised Open Information Extraction (2021.eacl-main)
Copied to clipboard
| Challenge: | Open Information Extraction (OIE) systems extract factual propositions into n-ary tuples . current datasets are limited in size and diversity . |
| Approach: | They propose to convert QA-SRL 2.0 dataset to large-scale OIE dataset LSOIE. |
| Outcome: | The proposed dataset is 20 times larger than the next largest human-annotated OIE dataset. |