| Challenge: | Data annotation is the process of labeling data that could be used to train machine learning models. |
| Approach: | They evaluate the performance of a large-scale language model developed by OpenAI . they compare it with traditional methods and analyze its output on a range of tasks . |
| Outcome: | The proposed model has shown impressive performance on a range of NLP tasks. |
Similar Papers
Want To Reduce Labeling Cost? GPT-3 Can Help (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Data annotation is labor-intensive and time-consuming for many NLP tasks. |
| Approach: | They propose to use GPT-3 to train models which are deployed for inference . they propose to combine pseudo labels from GPT3 with human labels . |
| Outcome: | The proposed method can be generalizable to many practical applications. |
Is GPT-4 a Good Data Analyst? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown their powerful capabilities in plenty of domains and tasks, including context understanding, code generation, language generation, data storytelling, etc. |
| Approach: | They propose to use GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains. |
| Outcome: | The proposed framework compares GPT-4 with human data analysts to perform end-to-end data analysis with databases from a wide range of domains. |
GPTs Are Multilingual Annotators for Sequence Generation Tasks (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods of data annotation are time-consuming and expensive . complexity of crowdsourcing increases when dealing with low-resource languages . |
| Approach: | They propose an autonomous method to gather unlabeled data and label them using large language models. |
| Outcome: | The proposed method is cost-efficient and applicable for low-resource language annotation. |
Large Language Models for Data Annotation and Synthesis: A Survey (2024.emnlp-main)
Copied to clipboard
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, Huan Liu
| Challenge: | Existing surveys focus on LLMs' specific utility for data annotation and synthesis. |
| Approach: | They propose to use large language models to generate annotations from raw data . they also propose to review learning strategies for models utilizing LLM-generated annotations . |
| Outcome: | The proposed models can be used to improve the efficacy of machine learning models by generating and labeling raw data with relevant information. |
Characterizing Human and Zero-Shot GPT-3.5 Object-Similarity Judgments (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in large language models have yielded few-shot, human-comparable performance on a range of tasks, but studies of LLM annotation accuracy and behavior are sparse. |
| Approach: | They characterize OpenAI’s GPT-3.5’s judgment on a behavioral task for implicit object categorization and give similarities and differences between them. |
| Outcome: | The proposed model augments human responses with LLMs for domains where data is sparse or compute resources are low. |
GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction (2024.acl-long)
Copied to clipboard
| Challenge: | Current benchmarks for social biases have limitations in scope, grounding, quality and human effort required. |
| Approach: | They propose to use a language model to help with the development of bias benchmarks . they extend previous work to a new community and set of biases: the Jewish community and antisemitism . |
| Outcome: | The proposed LLM does not perform well on the Jewish community and antisemitism task. |
LightTag: Text Annotation Platform (2021.emnlp-demo)
Copied to clipboard
| Challenge: | LightTag is a text annotation tool built on the premise of global optimization by addressing annotator as well as project managers and data scientists who manage the work and enforce production quality. |
| Approach: | They propose to use LightTag to optimize the global NLP process by addressing annotators as well as project managers and data scientists who manage the work and enforce production quality. |
| Outcome: | The proposed tool is based on the theory of constraints and is available for free for academic use. |
Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish (2022.lrec-1)
Copied to clipboard
Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey Öhman, Fredrik Carlsson, Magnus Sahlgren
| Challenge: | a prerequisite for building large-scale generative models for other languages is access to large amounts of high-quality text data and powerful computational resources. |
| Approach: | They present a 3.5 billion parameter autoregressive language model, trained on a 100 GB Swedish corpus. |
| Outcome: | The proposed model performs well on a 100 GB Swedish corpus and is competent in comparison with existing models of similar size. |
Corpus Considerations for Annotator Modeling and Scaling (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent trends in natural language processing and annotation tasks emphasize individual perspectives . annotator models that rely on a single ground truth may disregard valuable minority perspectives omissions . |
| Approach: | They propose a composite embedding approach to investigate annotator modeling techniques . they show that the commonly used user token model consistently outperforms more complex models . |
| Outcome: | The proposed model outperforms more complex models on a given dataset. |
Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape . established automatic evaluation metrics are poor surrogates, correlating weakly with human judgement. |
| Approach: | They propose to use both automatic and human evaluation to evaluate generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction. |
| Outcome: | The proposed model outperforms many popular models according to human reviewers on the majority of metrics, while scoring much worse when using classic automatic evaluation metrics. |