Unveiling Strengths and Weaknesses of NLP Systems Based on a Rich Evaluation Corpus: The Case of NER in French (2024.lrec-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is an applicative task for which annotation schemes vary . a lack of robustness of some tools towards textual variation limits evaluation . |
| Approach: | They propose a gold corpus for french annotated with a rich tagset that enables comparison with multiple annotation schemes. |
| Outcome: | The proposed framework enables a fair comparison of NER systems across textual genres and annotation schemes. |
Similar Papers
Comparing Annotated Datasets for Named Entity Recognition in English Literature (2022.lrec-1)
Copied to clipboard
| Challenge: | Generally speaking, the majority of NER tools struggle to perform well when the entities in the text contain specific characteristics. |
| Approach: | They analysed two existing annotated datasets and two additional gold standard datasets to evaluate the performance of two NER tools. |
| Outcome: | The results show that the performance of two NER tools varies significantly depending on the gold standard used for the individual evaluations. |
NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a widely adopted NLP task . authors present three variants of NER task, with dataset to support them . |
| Approach: | They propose three variants of the NER task, together with a dataset to support them . they propose a move towards more fine-grained entities and zero-shot recognition . |
| Outcome: | The proposed model matches or surpasses existing models in NER tasks . the proposed model is based on a large, silver-annotated corpus of 4 million paragraphs . |
What do we really know about State of the Art NER? (2022.lrec-1)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a well researched task and widely used in real world NLP scenarios. |
| Approach: | They perform a broad evaluation of Named Entity Recognition using a popular dataset that takes into consideration various text genres and sources constituting the dataset at hand. |
| Outcome: | The proposed models perform on a popular dataset and generate six new adversarial test sets through small perturbations in the original test set, replacing select entities while retaining the context. |
A Gold Standard for Multilingual Automatic Term Extraction from Comparable Corpora: Term Structure and Translation Equivalents (L18-1)
Copied to clipboard
| Challenge: | Terms are notoriously difficult to identify, both automatically and manually. |
| Approach: | They propose a method to annotate terms manually from a comparable corpus . they show that the gold standard provides a tool for evaluation and a rich source of information . |
| Outcome: | The proposed method provides a tool for evaluation and rich source of information about terms. |
Extrinsic Evaluation of French Dependency Parsers on a Specialized Corpus: Comparison of Distributional Thesauri (2020.lrec-1)
Copied to clipboard
| Challenge: | Using a frequency-based method, we can identify subsets of the same word contexts without any reference data. |
| Approach: | They compare 11 different French dependency parsers on a specialized corpus to generate distributional thesauri using a frequency-based method. |
| Outcome: | The proposed method can identify relevant subsets without reference data and the similarity is confirmed on a restricted distributional benchmark. |
Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark (2024.naacl-long)
Copied to clipboard
Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Suppa, Hila Gonen, Joseph Marvin Imperial, Börje Karlsson, Peiqin Lin, Nikola Ljubešić, Lester James Miranda, Barbara Plank, Arij Riabi, Yuval Pinter
| Challenge: | In named entity recognition, the majority of annotation efforts are centered on English, and cross-lingual transfer performance remains brittle. |
| Approach: | They propose to develop gold-standard named entity recognition benchmarks in many languages using a cross-lingual consistent schema. |
| Outcome: | The proposed benchmarks will be released to the public in 2022 . they will provide baselines on in-language and cross-lingual learning settings. |
Interpretable Multi-dataset Evaluation for Named Entity Recognition (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation methods for named entity recognition tasks are difficult to interpret . authors present a general methodology for interpretable evaluation for named entities . |
| Approach: | They propose a general methodology for interpretable evaluation for named entity recognition task. |
| Outcome: | The proposed evaluation method enables researchers to interpret differences in models and datasets . it makes it easy for future researchers to run similar analyses and drive progress in this area . |
HardEval: Focusing on Challenging Tokens to Assess Robustness of NER (2020.lrec-1)
Copied to clipboard
| Challenge: | Named entity recognition (NER) systems are often evaluated on human annotations . a new evaluation method focuses on subsets of tokens that represent specific sources of errors . |
| Approach: | They propose a method that focuses on subsets of tokens that represent specific sources of errors. |
| Outcome: | The proposed evaluation method focuses on subsets of tokens that represent specific sources of errors. |
UkraiNER: A New Corpus and Annotation Scheme towards Comprehensive Entity Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Named entity recognition excludes nested, discontinuous, non-named entities in practice . despite attempts to broaden their coverage, the most restrictive variant of NER remains the default . |
| Approach: | They propose a new annotation scheme that offers higher comprehensiveness while preserving simplicity. |
| Outcome: | The proposed scheme offers higher comprehensiveness while preserving simplicity . it also includes an annotation tool to implement the scheme on the corpus UkraiNER . |
CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English (2024.lrec-main)
Copied to clipboard
| Challenge: | a glass ceiling for named entity recognition systems has been suggested for 2021 . however, the performance of the most popular NER benchmarks has plateaued since then . we investigate what NER models are still struggling with . |
| Approach: | They perform a fine-grained evaluation of the model outputs by adding document annotations to the CoNLL-03 English dataset to identify lingering errors. |
| Outcome: | The proposed model is able to correct errors and guide future work. |