Understanding the Influence of Synthetic Data for Text Embedders (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in general purpose text embedders have been driven by training on synthetic training data. |
| Approach: | They propose to use GPT-4 to produce high quality synthetic data that expands existing training datasets for embeddings to new tasks. |
| Outcome: | The proposed dataset is high quality and leads to consistent improvements in performance. |
Similar Papers
Improving Text Embeddings with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for obtaining text embeddings require complex training pipelines . authors leverage proprietary LLMs to generate diverse synthetic data for text embeds based on 93 languages . |
| Approach: | They propose a method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. |
| Outcome: | The proposed method achieves strong performance on competitive text embedding benchmarks without using any labeled data. |
Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have explored using large language models to generate synthetic datasets . however, the effectiveness of the LLM-generated synthetic data is inconsistent across different classification tasks. |
| Approach: | They propose to use large language models to generate synthetic datasets to better understand factors that moderate the effectiveness of LLM-generated synthetic data. |
| Outcome: | The results show that subjectivity is negatively associated with the performance of the model trained on synthetic data. |
Synthetic Data in the Era of Large Language Models (2025.acl-tutorials)
Copied to clipboard
| Challenge: | 'synthetic data' is a data generated with the assistance of large language models to make dataset construction faster and cheaper. |
| Approach: | This tutorial seeks to build a shared understanding of recent progress in synthetic data generation from NLP and related fields by grouping and describing major methods, applications, and open problems. |
| Outcome: | This tutorial will describe methods, applications, and open problems that have been developed and are being used to improve the quality and efficiency of synthetic data generation. |
Text Classification with Few Examples using Controlled Generalization (N19-1)
Copied to clipboard
| Challenge: | Current training data for text classification is limited, resulting in limited generalization capacity. |
| Approach: | They propose a feed-forward network that can generalize from unlabeled parsed corpora to produce task-specific semantic vectors. |
| Outcome: | The proposed approach is especially effective in low-data scenarios compared to state-of-the-art methods. |
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) provide a data-centric solution to alleviate limitations of real-world data with synthetic data generation. |
| Approach: | They propose a generic workflow for LLM-driven synthetic data generation. |
| Outcome: | The proposed workflows highlight gaps in existing research and outline avenues for future studies. |
Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls (2025.emnlp-main)
Copied to clipboard
Feiyang Kang, Newsha Ardalani, Michael Kuchnik, Youssef Emad, Mostafa Elhoushi, Shubhabrata Sengupta, Shang-Wen Li, Ramya Raghavendra, Ruoxi Jia, Carole-Jean Wu
| Challenge: | a large-scale empirical study compares natural web data, diverse synthetic types, and mixtures of natural and synthetic data. |
| Approach: | They conduct a large-scale empirical study on large-volume LLMs using a unified protocol and scaling laws. |
| Outcome: | The proposed method is faster than pre-training on natural web data, the authors show . their results are consistent with previous studies on rephrased text and textbooks . |
How Well Do Text Embedding Models Understand Syntax? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text embedding models have not addressed syntactic understanding challenges, highlighting ineffectiveness and enhancing generalization ability. |
| Approach: | They propose to examine the ability of text embedding models to generalize across syntactic contexts. |
| Outcome: | The proposed models exhibit high similarity socres at this simple task. |
Little Giants: Synthesizing High-Quality Embedding Data at Scale (2025.naacl-long)
Copied to clipboard
| Challenge: | Synthetic data generation is an increasingly popular way of training models without the need for large, manually labeled datasets. |
| Approach: | They propose a framework that aligns open-source small models to efficiently generate large-scale embedding data. |
| Outcome: | The proposed framework outperforms state-of-the-art embedding models by using only 1/10 of the GPT API calls. |
BioReddit: Word Embeddings for User-Generated Biomedical NLP (D19-62)
Copied to clipboard
| Challenge: | a corpus of medical-themed posts was scrapped from Reddit to train word embeddings on downstream tasks. |
| Approach: | They propose to train word embeddings from a corpus of medical forums from reddit scrapping posts from medical-themed subreddits. |
| Outcome: | The proposed system outperforms embeddings trained on general purpose data or on scientific papers when applied on user-generated content. |
Text Embeddings Reveal (Almost) As Much As Text (2023.emnlp-main)
Copied to clipboard
| Challenge: | a vector database of dense text embeddings stores only the text data, not the original text . a multi-step method that iteratively corrects and re-embeds text can recover 92% of 32-token text inputs exactly. |
| Approach: | They propose a method that iteratively corrects and re-embeds text to recover 92% of 32-token text inputs exactly. |
| Outcome: | The proposed method recovers 92% of 32-token text inputs exactly. |