Papers by Pin-Jie Lin
Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning (2024.acl-srw)
Copied to clipboard
| Challenge: | Identifying beneficial tasks to transfer from is a critical step toward successful intermediate-task transfer learning. |
| Approach: | They propose a method that measures pairwise token similarity using maximum inner product search to improve task prediction. |
| Outcome: | The proposed method improves task prediction scores from 2.59% to 3.96% for tasks requiring reasoning abilities, but not for reasoning abilities. |
Revisiting Sample Size Determination in Natural Language Understanding (2023.findings-acl)
Copied to clipboard
Ernie Chang, Muhammad Hassan Rashid, Pin-Jie Lin, Changsheng Zhao, Vera Demberg, Yangyang Shi, Vikas Chandra
| Challenge: | Recent work has sought to reduce the annotation costs through the use of active learning and data sampling. |
| Approach: | They propose to estimate the training sample size needed to achieve a targeted model performance based on small amount of training samples. |
| Outcome: | The proposed approach predicts model performance within a small margin of mean absolute error (0.9%) with only 10% data. |
Scaling Parameter-Constrained Language Models with Quality Data (2024.emnlp-industry)
Copied to clipboard
Ernie Chang, Matteo Paltenghi, Yang Li, Pin-Jie Lin, Changsheng Zhao, Patrick Huber, Zechun Liu, Rastislav Rabatin, Yangyang Shi, Vikas Chandra
| Challenge: | Scaling laws in language modeling quantify training loss as a function of dataset size and model parameters, but neglect the critical role of data quality in model generalization. |
| Approach: | They propose to use effective training tokens as a combination of text diversity and syntheticity as measured by a teacher model to calculate scaling laws. |
| Outcome: | The proposed term effective training tokens is a combination of two readily-computed indicators of text diversity and syntheticity as measured by a teacher model. |
Efficient Model Development through Fine-tuning Transfer (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern large language models face a major bottleneck: each new version of a pre-trained model requires expensive and repetitive alignment. |
| Approach: | They propose a method that transfers fine-tuning updates across model versions . they extract the diff vector, which is the difference in parameters induced by fine-uning, from a source model and apply it to the base of a different target model. |
| Outcome: | The proposed method reduces training costs while maintaining model performance. |
Self-Vocabularizing Training for Neural Machine Translation (2025.naacl-srw)
Copied to clipboard
| Challenge: | Past vocabulary learning techniques identify relevant vocabulary before training, relying on corpus statistics or frequency counts without considering contextual information or the model's ability to represent it. |
| Approach: | They propose a method that self-vocabularizes a smaller, more optimal vocabulary by pairing source sentences with the model's predictions to define a new vocabulary. |
| Outcome: | The proposed method produces a 1.49 BLEU improvement in the simulated model and an increase in unique token usage and a 6–8% reduction in vocabulary size. |
Target-Aware Language Modeling via Granular Data Sampling (2024.emnlp-main)
Copied to clipboard
Ernie Chang, Pin-Jie Lin, Yang Li, Changsheng Zhao, Daeil Kim, Rastislav Rabatin, Zechun Liu, Yangyang Shi, Vikas Chandra
| Challenge: | Language model pretraining is the cornerstone of universal language models (LMs), creating generalpurpose representations to excel across a variety of downstream tasks. |
| Approach: | They propose to use multi-granular tokens to sample large-scale language models for domain-specific use cases. |
| Outcome: | The proposed model outperforms random sampled samples on eight benchmarks with 1% of the data and performs on par with the full RefinedWeb data. |
Modeling Orthographic Variation Improves NLP Performance for Nigerian Pidgin (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets for Nigerian Pidgin are characterised by noise in the form of orthographic variations. |
| Approach: | They propose a phonetic-theoretic framework to generate orthographic variations to augment training data. |
| Outcome: | The proposed framework improves machine translation and sentiment analysis by combining real and synthesized orthographic variations. |