Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Cost (2024.lrec-main)
Copied to clipboard
| Challenge: | Current foundation models have shown impressive performance across various tasks, but they are not effective for everyone due to the imbalanced geographical and economic representation of the data used in the training process. |
| Approach: | They propose to identify the data to be annotated to balance model performance and annotation costs by finding countries with visual similarity for the topics. |
| Outcome: | The proposed methods improve model performance and reduce annotation costs by using data from countries with higher visual similarity for these topics. |
Similar Papers
Geo-Cultural Representation and Inclusion in Language Technologies (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | audi et al.: training and evaluation of language models rely on semi-structured data that is annotated by humans . e-learning tools do not integrate rich and diverse community perspectives into language technologies . |
| Approach: | They will examine how different socio-cultural perspectives influence what is taken as ground truth by models. |
| Outcome: | This tutorial examines how different socio-cultural perspectives influence representations of global concepts. |
Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that multi-annotator datasets can improve performance when they expand from a single annotation per instance to multiple annotations. |
| Approach: | They propose a multi-annotator simulation process to generate datasets with varying annotation budgets and compare them to a single annotation per instance. |
| Outcome: | The proposed model can generate datasets with varying annotation budgets and show that similar datasets can lead to varying performance gains. |
Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | In subjective tasks, the inclusion of diverse annotators is crucial as their unique perspectives significantly influence the annotations. |
| Approach: | They propose a framework that minimizes the annotation budget while maximizing the predictive performance for each annotator. |
| Outcome: | The proposed framework surpasses the previous SOTA in capturing the annotators’ individual perspectives with as little as 25% of the original annotation budget on two datasets. |
The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection (2025.findings-naacl)
Copied to clipboard
Tomáš Horych, Christoph Mandl, Terry Ruas, Andre Greiner-Petter, Bela Gipp, Akiko Aizawa, Timo Spinde
| Challenge: | Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. |
| Approach: | They propose to use Large Language Models to automate annotation process and train classifiers on large datasets. |
| Outcome: | The proposed model outperforms all of the annotator LLMs on two media bias benchmark datasets (BABE and BASIL) while maintaining data quality. |
Bridging the Digital Divide: Performance Variation across Socio-Economic Factors in Vision-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Among the minority groups under-represented in AI, data from low-income households are often overlooked in data collection and model evaluation. |
| Approach: | They evaluate the performance of a vision-language model on a geo-diverse dataset . they highlight insights that can help mitigate these issues and propose actionable steps for economic-level inclusive AI development. |
| Outcome: | The proposed model performs lower for the poorer groups than the wealthier groups across topics and countries. |
Corpus Considerations for Annotator Modeling and Scaling (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent trends in natural language processing and annotation tasks emphasize individual perspectives . annotator models that rely on a single ground truth may disregard valuable minority perspectives omissions . |
| Approach: | They propose a composite embedding approach to investigate annotator modeling techniques . they show that the commonly used user token model consistently outperforms more complex models . |
| Outcome: | The proposed model outperforms more complex models on a given dataset. |
Is it worth it? Budget-related evaluation metrics for model selection (L18-1)
Copied to clipboard
| Challenge: | linguistic resources can be labor-intensive, requiring great amounts of work-hours and expert annotation. |
| Approach: | They propose a machine learning model that pre-annotates or filters content before annotating it . they argue that the model with the highest F-score may not have best separation . |
| Outcome: | a case study shows that the model with the highest F-score does not yield the highest profits . the model that has the highest score does not produce the highest profit, the study shows . |
Pre-train or Annotate? Domain Adaptation with a Constrained Budget (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that pre-training in-domain language models can boost performance when adapting to a new domain. |
| Approach: | They propose to combine annotation and pre-training to maximize performance under budget constraints. |
| Outcome: | The proposed approach is based on the annotation cost of three procedural text datasets and pre-training cost of 3 in-domain language models. |
Scaling Cultural Resources for Improving Generative Models (2026.findings-eacl)
Copied to clipboard
Hayk Stepanyan, Aishwarya Verma, Andrew Zaldivar, Rutledge Chin Feman, Erin MacMurray van Liemt, Charu Kalia, Vinodkumar Prabhakaran, Sunipa Dev
| Challenge: | generative models have been known to have reduced performance in different global cultural contexts and languages. |
| Approach: | They construct a pipeline to collect and contribute culturally salient, multilingual data . they argue such data can assess the state of the global applicability of generative AI models . |
| Outcome: | The proposed pipeline can assess the state of the global applicability of our models and improve upon cross-cultural gaps. |
You Are What You Annotate: Towards Better Models through Annotator Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Annotator disagreement is ubiquitous in natural language processing tasks. |
| Approach: | They propose to model annotators' idiosyncrasies and account for their idioms by creating representations for each annotator and their annotations. |
| Outcome: | The proposed model improves on an existing dataset with eight annotators with inherent disagreements while increasing model size by 1%. |