GPTs Are Multilingual Annotators for Sequence Generation Tasks (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods of data annotation are time-consuming and expensive . complexity of crowdsourcing increases when dealing with low-resource languages . |
| Approach: | They propose an autonomous method to gather unlabeled data and label them using large language models. |
| Outcome: | The proposed method is cost-efficient and applicable for low-resource language annotation. |
Similar Papers
Want To Reduce Labeling Cost? GPT-3 Can Help (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Data annotation is labor-intensive and time-consuming for many NLP tasks. |
| Approach: | They propose to use GPT-3 to train models which are deployed for inference . they propose to combine pseudo labels from GPT3 with human labels . |
| Outcome: | The proposed method can be generalizable to many practical applications. |
Crowdsourcing Natural Language Data at Scale: A Hands-On Tutorial (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | a tutorial on crowdsourcing for efficient data annotation will introduce crowdsourcing and provide an overview of the technology. |
| Approach: | This tutorial will introduce users to efficient data annotation via crowdsourcing marketplaces. |
| Outcome: | This tutorial will introduce users to the use of crowdsourcing for data annotation. |
Large Language Models for Data Annotation and Synthesis: A Survey (2024.emnlp-main)
Copied to clipboard
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, Huan Liu
| Challenge: | Existing surveys focus on LLMs' specific utility for data annotation and synthesis. |
| Approach: | They propose to use large language models to generate annotations from raw data . they also propose to review learning strategies for models utilizing LLM-generated annotations . |
| Outcome: | The proposed models can be used to improve the efficacy of machine learning models by generating and labeling raw data with relevant information. |
Is GPT-3 a Good Data Annotator? (2023.acl-long)
Copied to clipboard
| Challenge: | Data annotation is the process of labeling data that could be used to train machine learning models. |
| Approach: | They evaluate the performance of a large-scale language model developed by OpenAI . they compare it with traditional methods and analyze its output on a range of tasks . |
| Outcome: | The proposed model has shown impressive performance on a range of NLP tasks. |
Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | Developing a theory of crowdsourcing for practical language problems remains an open challenge . |
| Approach: | This tutorial exposes NLP researchers to data collection crowdsourcing methods and principles through case studies. |
| Outcome: | This tutorial exposes NLP researchers to various data collection crowdsourcing methods and practices through case studies. |
Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency (2024.lrec-main)
Copied to clipboard
Toyin D. Aguda, Suchetha Siddagangappa, Elena Kochkina, Simerjot Kaur, Dongsheng Wang, Charese Smiley
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable performance in data annotation tasks on general domain datasets, but their effectiveness on domain specific datasets remains under-explored. |
| Approach: | They compare the annotations produced by three LLMs against expert annotators and crowdworkers. |
| Outcome: | The proposed models outperform expert crowdworkers and crowd-sourced annotators on domain specific datasets. |
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (D19-1)
Copied to clipboard
| Challenge: | Having only a few workers generate the majority of dataset examples raises concerns about data diversity . |
| Approach: | They perform a series of experiments to investigate annotator biases in recent NLU datasets . they find that models are able to recognize the most productive annotators . |
| Outcome: | The results show that models can recognize the most productive annotators and do not generalize well to examples from annotator that did not contribute to the training set. |
Crowd-sourcing annotation of complex NLU tasks: A case study of argumentative content annotation (D19-59)
Copied to clipboard
| Challenge: | Recent advances in machine reading and listening comprehension involve the annotation of long texts. |
| Approach: | They propose a way to perform a sentence-by-sentence annotation task with crowd annotators. |
| Outcome: | The proposed approach can be used to identify claims in a debate speech. |
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability. |
| Approach: | They propose a data augmentation technique that leverages large-scale language models to generate real text samples from a mixture of real samples. |
| Outcome: | The proposed method outperforms existing methods on diverse classification tasks. |
GPT-NER: Named Entity Recognition via Large Language Models (2025.findings-naacl)
Copied to clipboard
Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, Guoyin Wang, Chen Guo
| Challenge: | Large-scale language models (LLMs) have shown impressive ability for in-context learning with limited training data. |
| Approach: | They propose a novel sequence labeling task that transforms a sequence labeled as a text-generation task into a self-verification task that LLMs can adapt to. |
| Outcome: | The proposed model performs better on NER than supervised models on a variety of tasks . the proposed model can be easily adapted by LLMs to generate a text sequence . |