Evaluating D-MERIT of Partial-annotation on Information Retrieval (2024.emnlp-main)
Copied to clipboard
Royi Rassin, Yaron Fairstein, Oren Kalinsky, Guy Kushilevitz, Nachshon Cohen, Alexander Libov, Yoav Goldberg
| Challenge: | Using partially-annotated datasets for evaluation can lead to false conclusions . a dataset containing only a subset of relevant passages might result in misleading rankings . |
| Approach: | They propose to use a Wikipedia passage retrieval evaluation set to contain all relevant passages for each query. |
| Outcome: | The proposed dataset can be downloaded from https://d-merit.github.io. |
Similar Papers
On the Limitations of Reference-Free Evaluations of Generated Text (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has shown that evaluation metrics which accurately estimate the quality of generated text are limited in their ability to evaluate generated text. |
| Approach: | They argue that reference-free metrics are limited in their ability to evaluate generated text . they recommend that they be used as diagnostic tools for analyzing and understanding model behavior . |
| Outcome: | The proposed evaluation metrics are limited in their ability to evaluate generated text . they can be optimized at test time, can be biased against models with similar outputs . |
Open-World Evaluation for Retrieving Diverse Perspectives (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing retrieval systems only cover diverse perspectives on 33.74% of the examples . existing systems only focus on relevance to the question, ignoring diversity. |
| Approach: | They build a Benchmark for Retrieval Diversity for Subjective questions (BERDS) based on a question and diverse perspectives associated with the question . they evaluate retrievers paired with a corpus to determine whether each document contains a perspective . |
| Outcome: | The proposed approach improves retrieval diversity on complex questions . existing retrieval systems only cover diverse perspectives on 33.74% of the examples . |
Is a Document Educational or Just Wikipedia-Style? — Pitfalls of Classifier-Based Quality Filtering (2026.acl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are pre-trained on massive data corpora, and the quality of these corporales is one of the main factors in achieving stateof-the-art performance. |
| Approach: | They propose to use Wikipedia-style reformatting to alter a model's quality assessment and enable low-quality content to surpass filtering thresholds. |
| Outcome: | The proposed model would reverse filtering decision for approximately 7% of evaluated documents, thereby admitting content into the pre-training corpus that would otherwise have been excluded. |
Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED (2022.acl-long)
Copied to clipboard
| Challenge: | Document-level relation extraction is a challenging task as it requires reasoning across multiple sentences. |
| Approach: | They propose to use a recommend-revise scheme to reduce the workload of annotators by providing them with candidate relation instances from distant supervision to supplement and remove relational facts. |
| Outcome: | The proposed dataset is the first large-scale and human-annotated dataset for relation extraction. |
Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using incomplete annotations, we find that false negative samples are prevalent in the DocRED dataset . we reannotate 4,053 documents in the dataset by adding the missed relation triples back to the original DocRED. |
| Approach: | They propose to re-annotate 4,053 documents in the document-level relation extraction dataset by adding missing relation triples back to the original DocRED. |
| Outcome: | The proposed dataset improves on the existing DocRED dataset by 13 F1 points. |
A Comparison Of Emotion Annotation Schemes And A New Annotated Data Set (L18-1)
Copied to clipboard
| Challenge: | a series of study on positive/negative sentiments has been conducted on tweets, but recognition of more nuanced affect has received little attention . valence, arousal, dominance and surprise are the most commonly used emotion representation schemes . |
| Approach: | They propose to annotate tweets with scores on four emotion dimensions . they compare annotator agreement with relative annotation schemes over categorical ones . |
| Outcome: | The proposed model improves agreement with relative annotation schemes over categorical ones on Ekman's six basic emotions. |
Re3val: Reinforced and Reranked Generative Retrieval (2024.findings-eacl)
Copied to clipboard
| Challenge: | generative retrieval models encode pointers to information in a corpus as an index within the model’s parameters. |
| Approach: | They propose a generative retrieval model that leverages contextual information to rerank retrieved page titles and utilizes REINFORCE to maximize rewards generated by constrained decoding. |
| Outcome: | The proposed model can't be tuned for the downstream readers as decoding the page title is a non-differentiable operation. |
Unveiling Strengths and Weaknesses of NLP Systems Based on a Rich Evaluation Corpus: The Case of NER in French (2024.lrec-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is an applicative task for which annotation schemes vary . a lack of robustness of some tools towards textual variation limits evaluation . |
| Approach: | They propose a gold corpus for french annotated with a rich tagset that enables comparison with multiple annotation schemes. |
| Outcome: | The proposed framework enables a fair comparison of NER systems across textual genres and annotation schemes. |
A Short Survey on Sense-Annotated Corpora (2020.lrec-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a key task in Natural Language Understanding. |
| Approach: | They propose to use sense-annotated corpora for supervised Word Sense Disambiguation. |
| Outcome: | The proposed methods have been compared with knowledge-based approaches and have shown to be more efficient when they are available. |
Corpus Considerations for Annotator Modeling and Scaling (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent trends in natural language processing and annotation tasks emphasize individual perspectives . annotator models that rely on a single ground truth may disregard valuable minority perspectives omissions . |
| Approach: | They propose a composite embedding approach to investigate annotator modeling techniques . they show that the commonly used user token model consistently outperforms more complex models . |
| Outcome: | The proposed model outperforms more complex models on a given dataset. |