Evidence > Intuition: Transferability Estimation for Encoder Selection (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on LM transferability have focused on a priori tuning of encoders . prior work has examined the different yet related tasks of performance prediction . |
| Approach: | They propose to generate quantitative evidence to predict which LM will perform best on a target task without fine-tuning all candidates. |
| Outcome: | The proposed model outperforms the standard of human practitioner ranking in 94% of the setups. |
Similar Papers
Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for NLP tasks require fine-tuning, but it is computationally infeasible. |
| Approach: | They propose an approach that inexpensively estimates a ranking of the expected performance of a given set of transformer language models for a specific task. |
| Outcome: | The proposed model improves the Pearson correlation coefficient between the true model ranks and the estimate. |
How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Transferability estimation has been a topic of great interest in computer vision fields . a lack of a comprehensive comparison between these estimation methods is a problem . |
| Approach: | They conduct a thorough survey of existing methods to find the most suitable model . they also outline difficulties of consideration of training details and applicability to text generation . |
| Outcome: | The proposed methods perform well with superiorities in effectiveness and efficiency. |
Exploring and Predicting Transferability across NLP Tasks (2020.emnlp-main)
Copied to clipboard
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, Mohit Iyyer
| Challenge: | Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. |
| Approach: | They conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems. |
| Outcome: | The proposed model can improve performance even with low-data source tasks that differ substantially from the target task. |
Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Ranking (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for text ranking are based on intuition, but their estimated transferability may not align well with the objectives of text ranking. |
| Approach: | They propose to compute expected rank as transferability, explicitly reflecting the model’s ranking capability. |
| Outcome: | The proposed method shows significant improvements over previous classification-oriented TE methods, human intuition, and ChatGPT with minor time consumption. |
Quantifying Adaptability in Pre-trained Language Models with 500 Tasks (2022.naacl-main)
Copied to clipboard
| Challenge: | a recent study examines the features and limits of LM adaptability to new tasks . many questions about the nature and limits remain unanswered . |
| Approach: | They evaluate adaptability to new tasks using a new benchmark, TaskBench500 . they find adaptation procedures differ dramatically in their ability to memorize small datasets . |
| Outcome: | The proposed benchmark compares 500 procedurally generated sequence modeling tasks to a new benchmark. |
Less is More: Parameter-Efficient Selection of Intermediate Tasks for Transfer Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Prior methods producing useful task rankings are infeasible for large source pools . Embedding space maps (ESMs) reduce execution time and disk space usage . |
| Approach: | They introduce Embedded Space Maps (ESMs) that approximate the effect of fine-tuning a language model. |
| Outcome: | The proposed method reduces execution time and disk space usage by 10 and 278, respectively, while retaining high selection performance. |
Linguistic Knowledge and Transferability of Contextual Representations (N19-1)
Copied to clipboard
| Challenge: | Recent work has explored contextual word representations, which assign each word a vector that is a function of the entire input sequence. |
| Approach: | They compare pretrained word representations with 16 diverse probing tasks to examine their transferability. |
| Outcome: | The pretrained representations are successful across a diverse set of NLP tasks . the models are competitive with state-of-the-art models but fail on fine-grained tasks requiring fine-granular knowledge, the study finds . |
Choosing Transfer Languages for Cross-Lingual Learning (P19-1)
Copied to clipboard
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, Graham Neubig
| Challenge: | Cross-lingual transfer is a useful tool for improving performance of natural language processing (NLP) on low-resource languages. |
| Approach: | They propose to use cross-lingual transfer to improve accuracy of low-resource languages . they build models that consider features to perform prediction on such languages based on ranking problem . |
| Outcome: | The proposed model predicts good transfer languages much better than baselines considering single features in isolation. |
Towards Instance-Level Parser Selection for Cross-Lingual Transfer of Dependency Parsers (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods of cross-lingual parser transfer focus on predicting the best parsers for a low-resource target language globally. |
| Approach: | They propose a cross-lingual parser transfer paradigm that uses instance-level parsers to predict the best parsing for a target language at treebank level. |
| Outcome: | The proposed model outperforms existing models on 13/20 and 14/20 test languages. |
Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies focus on data selection but lack a clear, unified framework . variability in experimental settings complicates systematic comparisons . |
| Approach: | They propose a three-stage scheme to standardize data selection for fine-tuning large language models . they propose unified comparison approach that incorporates ratio-based efficiency and ranking-based feasibility metrics to address inconsistencies across experiments. |
| Outcome: | The proposed scheme outperforms existing methods in a dozen key studies and identifies key challenges. |