| Challenge: | obtaining training data is often the most difficult part of an NLP or ML project . obtaining data is important to estimate how much training data a system will require to achieve a target accuracy. |
| Approach: | They propose a performance extrapolation task to evaluate extrapolations on larger training sets. |
| Outcome: | The proposed method can predict accuracy on larger training datasets. |
Similar Papers
Predicting Fine-tuned Performance on Larger Datasets Before Creating Them (2025.coling-industry)
Copied to clipboard
| Challenge: | a method to estimate the performance of pretrained models fine-tuned with a larger dataset is proposed . a recent study found that fine-timing PMs with the small amount of data does not always result in ideal performance. |
| Approach: | They propose a method to estimate the performance of pretrained models fine-tuned with a larger dataset from the result with fewer epochs. |
| Outcome: | The proposed method can help resource-limited companies develop machine-learning models . it shows that when a model is fine-tuned with a larger dataset, its classification performance increases . |
Revisiting Sample Size Determination in Natural Language Understanding (2023.findings-acl)
Copied to clipboard
Ernie Chang, Muhammad Hassan Rashid, Pin-Jie Lin, Changsheng Zhao, Vera Demberg, Yangyang Shi, Vikas Chandra
| Challenge: | Recent work has sought to reduce the annotation costs through the use of active learning and data sampling. |
| Approach: | They propose to estimate the training sample size needed to achieve a targeted model performance based on small amount of training samples. |
| Outcome: | The proposed approach predicts model performance within a small margin of mean absolute error (0.9%) with only 10% data. |
On the data requirements of probing (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to probe neural networks are expensive and require large datasets. |
| Approach: | They propose a method to estimate the required number of data samples in probing datasets . they use a classification task to encode a text with a deep neural network . |
| Outcome: | The proposed method estimates the required number of data samples in two probing configurations and proves it is statistically powerful. |
Location Attention for Extrapolation to Longer Sequences (2020.acl-main)
Copied to clipboard
| Challenge: | Neural networks are surprisingly good at interpolating, but they are often unable to extrapolate patterns beyond the seen data. |
| Approach: | They propose to use a special type of extrapolation for natural language processing to generalize to sequences that are longer than the training ones. |
| Outcome: | The proposed model is more likely to extrapolate than models with common attention mechanisms. |
Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Instruction tuning language models can be expensive and expensive to train . current methods require extensive training on large datasets, resulting in high training costs. |
| Approach: | They propose a novel approach to selecting training data based on the learning percentage of the samples. |
| Outcome: | The proposed model performs better on models ranging from 1B to 13B in size compared to training on the entire dataset. |
Let’s Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | *Data Synthesis* is a promising way to train a small model with very little labeled data. |
| Approach: | They propose a framework that iteratively extrapolates the errors of a small model trained on a real-world validation dataset using a large language model. |
| Outcome: | The proposed framework reduces the gap between the synthesized dataset and the real data . it improves on multiple NLP tasks and on large models with human-annotated data. |
On the Importance of Data Size in Probing Fine-tuned Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. |
| Approach: | They propose to investigate the reasons behind the effectiveness of fine-tuning by examining the impact of data size on the extent of encoded linguistic knowledge. |
| Outcome: | The proposed probes show that the size of the training data affects the recoverability of the changes made to the model’s linguistic knowledge. |
Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models such as RoBERTa, ELECTRA, and T5 have a large number of parameters which makes them slow and computationally expensive. |
| Approach: | They propose a technique that utilizes a collection of models of varying capacities to accurately yet efficiently output predictions. |
| Outcome: | The proposed technique saves up to 88.93% computation cost and consistently achieves superior prediction accuracy with an improvement of up to 2.18%. |
Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using low-resource languages, we assess the quality of uncertainty estimates from a wide array of approaches, but with more data. |
| Approach: | They train models on sub-sampled datasets in three different languages to assess the confidence of a neural classifier. |
| Outcome: | The proposed models train on sub-sampled datasets in three different languages and show that the quality of uncertainty estimates suffers with more data. |
Predicting generalization performance with correctness discriminators (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models estimate accuracy of models on unlabeled test data, but they hide their own uncertainty. |
| Approach: | They propose a model that establishes upper and lower bounds on the accuracy without requiring gold labels for the unseen data. |
| Outcome: | The proposed model establishes upper and lower bounds on accuracy without requiring gold labels for the unseen data. |