Finding Spoken Identifications: Using GPT-4 Annotation for an Efficient and Fast Dataset Creation Pipeline (2024.lrec-main)
Copied to clipboard
Maliha Jahan, Helin Wang, Thomas Thebaud, Yinglun Sun, Giang Ha Le, Zsuzsanna Fagyal, Odette Scharenborg, Mark Hasegawa-Johnson, Laureano Moro Velazquez, Najim Dehak
| Challenge: | Existing datasets that are limited to a few dialects, ethnicities, and age groups are not annotated considering these factors. |
| Approach: | They propose a semi-automated dataset creation pipeline that leverages large language models to perform two complex annotation tasks using human annotations as ground truths. |
| Outcome: | The proposed pipeline reduces time required for the filtering and tagging tasks while losing no important information. |
Similar Papers
Rapidly Developing High-quality Instruction Data and Evaluation Benchmark for Large Language Models with Minimal Human Effort: A Case Study on Japanese (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have aimed to refine their capacity to accurately follow human instructions and navigate intricate scenarios. |
| Approach: | They propose a method that uses a set of instructions to translate English into Japanese and then generates Japanese instruction data using GPT-4. |
| Outcome: | The proposed method outperforms Japanese-Alpaca models in the evaluation benchmarks without human references. |
GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction (2024.acl-long)
Copied to clipboard
| Challenge: | Current benchmarks for social biases have limitations in scope, grounding, quality and human effort required. |
| Approach: | They propose to use a language model to help with the development of bias benchmarks . they extend previous work to a new community and set of biases: the Jewish community and antisemitism . |
| Outcome: | The proposed LLM does not perform well on the Jewish community and antisemitism task. |
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (D19-1)
Copied to clipboard
| Challenge: | Having only a few workers generate the majority of dataset examples raises concerns about data diversity . |
| Approach: | They perform a series of experiments to investigate annotator biases in recent NLU datasets . they find that models are able to recognize the most productive annotators . |
| Outcome: | The results show that models can recognize the most productive annotators and do not generalize well to examples from annotator that did not contribute to the training set. |
GPTs Are Multilingual Annotators for Sequence Generation Tasks (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods of data annotation are time-consuming and expensive . complexity of crowdsourcing increases when dealing with low-resource languages . |
| Approach: | They propose an autonomous method to gather unlabeled data and label them using large language models. |
| Outcome: | The proposed method is cost-efficient and applicable for low-resource language annotation. |
Can LLMs Augment Low-Resource Reading Comprehension Datasets? Opportunities and Challenges (2024.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive zero-shot performance on a wide range of NLP tasks. |
| Approach: | They propose to use large language models to augment extractive reading comprehension datasets by fine-tuning their annotations and comparing their performance to human annotators. |
| Outcome: | The proposed model can be used to augment extractive reading comprehension datasets. |
Artie Bias Corpus: An Open Dataset for Detecting Demographic Bias in Speech Applications (2020.lrec-1)
Copied to clipboard
| Challenge: | A speech technology exhibits demographic bias when performance is worse for one demographic group relative to another. |
| Approach: | They create an English dataset of expert-validated audio, transcript> pairs with demographic tags for age, gender, accent and open software which may be used to detect demographic bias in Automatic Speech Recognition systems. |
| Outcome: | The Artie Bias Corpus is a curated subset of the Mozilla Common Voice corpus, which is released under a Creative Commons CC0 license . |
Corpus Considerations for Annotator Modeling and Scaling (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent trends in natural language processing and annotation tasks emphasize individual perspectives . annotator models that rely on a single ground truth may disregard valuable minority perspectives omissions . |
| Approach: | They propose a composite embedding approach to investigate annotator modeling techniques . they show that the commonly used user token model consistently outperforms more complex models . |
| Outcome: | The proposed model outperforms more complex models on a given dataset. |
Large Language Models for Data Annotation and Synthesis: A Survey (2024.emnlp-main)
Copied to clipboard
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, Huan Liu
| Challenge: | Existing surveys focus on LLMs' specific utility for data annotation and synthesis. |
| Approach: | They propose to use large language models to generate annotations from raw data . they also propose to review learning strategies for models utilizing LLM-generated annotations . |
| Outcome: | The proposed models can be used to improve the efficacy of machine learning models by generating and labeling raw data with relevant information. |
Biases in Large Language Model-Elicited Text: A Case Study in Natural Language Inference (2025.coling-main)
Copied to clipboard
| Challenge: | Creating NLP datasets with Large Language Models (LLMs) is an attractive alternative to relying on crowd-source workers. |
| Approach: | They recreate a portion of the Stanford Natural Language Inference corpus using GPT-4, Llama-2 70b for Chat, and Mistral 7b Instruct. |
| Outcome: | The proposed model can be used to generate NLP datasets with stereotypical biases and annotation artifacts. |
Information Extraction from Legal Wills: How Well Does GPT-4 Do? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using information extraction from legal wills is an important application of artificial intelligence (AI) |
| Approach: | They propose a manually annotated dataset for Information Extraction (IE) from legal wills . they also use it to evaluate the performance of large language models (LLMs) |
| Outcome: | The proposed dataset can be used to evaluate large language models on IE from legal wills . it shows that the model performs reasonably well, but inconsistent outputs and overgeneralization are observed . |