CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English (2024.lrec-main)
Copied to clipboard
| Challenge: | a glass ceiling for named entity recognition systems has been suggested for 2021 . however, the performance of the most popular NER benchmarks has plateaued since then . we investigate what NER models are still struggling with . |
| Approach: | They perform a fine-grained evaluation of the model outputs by adding document annotations to the CoNLL-03 English dataset to identify lingering errors. |
| Outcome: | The proposed model is able to correct errors and guide future work. |
Similar Papers
CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models achieve F1-scores comparable to or exceed noise level in CoNLL-03 . current models have significant annotation errors, incompleteness, and inconsistencies in the data . |
| Approach: | They propose to add a layer of entity linking annotation to the CoNLL-03 corpus to correct 7.0% of all labels. |
| Outcome: | The proposed approach corrects 7.0% of all labels in the English CoNLL-03 dataset. |
Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023? (2023.acl-long)
Copied to clipboard
| Challenge: | NER models trained on 20-year-old test set may not perform well on modern data. |
| Approach: | They evaluate the generalization of over 20 different models trained on the CoNLL-2003 dataset . they find no evidence of performance degradation in pre-trained Transformers . |
| Outcome: | The proposed model generalizations show that some models generalize well on new data while others do not. |
Do “English” Named Entity Recognizers Work Well on Global Englishes? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Most of English named entity recognition datasets contain American or British English data . multiple problems may occur in low-resource English contexts, such as confusion of named entities with regionspecific meanings . |
| Approach: | They build a newswire dataset to analyze NER model performance on low-resource English variants . they find that models trained on the CoNLL or OntoNotes datasets experienced significant performance drops . |
| Outcome: | The results show that models trained on the CoNLL or OntoNotes datasets experienced significant performance drops. |
MultiCoNER v2: a Large Multilingual dataset for Fine-grained and Noisy Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a core task in Natural Language Processing. |
| Approach: | They present a dataset for fine-grained Named Entity Recognition covering 33 entity classes across 12 languages in monolingual and multilingual settings. |
| Outcome: | The proposed dataset covers 33 entity classes across 12 languages in monolingual and multilingual settings. |
Why Aren’t We NER Yet? Artifacts of ASR Errors in Named Entity Recognition in Spontaneous Speech Transcripts (2023.acl-long)
Copied to clipboard
Piotr Szymański, Lukasz Augustyniak, Mikolaj Morzy, Adrian Szymczak, Krzysztof Surdyk, Piotr Żelasko
| Challenge: | despite advances in language models, the transcript of spontaneous human-human conversations remains an insurmountable challenge for most models. |
| Approach: | They examine the relationship between ASR and NER errors which limit NER models' ability to recover entity mentions from spontaneous speech transcripts. |
| Outcome: | The proposed model fails even if no word errors are introduced by the ASR . the proposed model's performance deteriorates when applied to the ASL outputs . |
NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a widely adopted NLP task . authors present three variants of NER task, with dataset to support them . |
| Approach: | They propose three variants of the NER task, together with a dataset to support them . they propose a move towards more fine-grained entities and zero-shot recognition . |
| Outcome: | The proposed model matches or surpasses existing models in NER tasks . the proposed model is based on a large, silver-annotated corpus of 4 million paragraphs . |
Let’s Stop Incorrect Comparisons in End-to-end Relation Extraction! (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature on Relation Extraction (RE) uses multiple evaluation setups to compare performance. |
| Approach: | They propose to quantify the most common comparison mistake and evaluate it leads to overestimating the final RE performance by around 5% on ACE05. |
| Outcome: | The proposed meta-analysis overestimates the final RE performance by around 5% on ACE05. |
Comparing Annotated Datasets for Named Entity Recognition in English Literature (2022.lrec-1)
Copied to clipboard
| Challenge: | Generally speaking, the majority of NER tools struggle to perform well when the entities in the text contain specific characteristics. |
| Approach: | They analysed two existing annotated datasets and two additional gold standard datasets to evaluate the performance of two NER tools. |
| Outcome: | The results show that the performance of two NER tools varies significantly depending on the gold standard used for the individual evaluations. |
NNE: A Dataset for Nested Named Entity Recognition in English Newswire (P19-1)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is widely used in downstream tasks but most tools focus on flat mention structure over coarse schemas. |
| Approach: | They describe a fine-grained, nested named entity dataset over the Wall Street Journal portion of the Penn Treebank. |
| Outcome: | The proposed dataset comprises 279,795 mentions of 114 entity types with up to 6 layers of nesting. |
What do we really know about State of the Art NER? (2022.lrec-1)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a well researched task and widely used in real world NLP scenarios. |
| Approach: | They perform a broad evaluation of Named Entity Recognition using a popular dataset that takes into consideration various text genres and sources constituting the dataset at hand. |
| Outcome: | The proposed models perform on a popular dataset and generate six new adversarial test sets through small perturbations in the original test set, replacing select entities while retaining the context. |