Papers by Sheng Zha
Meta-learning via Language Model In-context Tuning (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have reduced "task learning and prediction" to a simple sequence prediction problem. |
| Approach: | They propose a meta-learning method that recasts task adaptation and prediction as a sequence prediction problem. |
| Outcome: | The proposed method outperforms MAML on two classification tasks and improves on binaryClfs. |
Fine-tuning Language Models for Joint Rewriting and Completion of Code with Potential Bugs (2024.findings-acl)
Copied to clipboard
| Challenge: | Previous work has demonstrated shortcomings of large language models of code (CodeLLMs) in completing drafty partial code with potential bugs. |
| Approach: | They propose to use large language models of code to fine-tune their models to rewrite and complete drafty partial code into functional full programs. |
| Outcome: | The proposed approach achieves superior pass rates over baselines and preserves the integrity of the original partial implementations. |
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. |
| Approach: | They aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning by increasing the number of tasks and incorporating smaller sets of related tasks. |
| Outcome: | The proposed model improves on unseen target tasks by increasing the scale of multi-task learning to incorporate more tasks and developing similarity metrics to incorporate tasks related to the target task. |
Extreme Miscalibration and the Illusion of Adversarial Robustness (2024.acl-long)
Copied to clipboard
| Challenge: | emergence of the Adversarial Training paradigm has shown some success in training models to be more robust to these small adversarial perturbations. |
| Approach: | They propose to use adversarial examples to detect adversarials by miscalibrating models to mask gradients in a way that interferes with adversarial attack search methods. |
| Outcome: | The proposed model gains are an illusion of robustness (IOR) and urge the NLP community to incorporate test-time temperature scaling into their robustness evaluations to ensure that any observed gains are genuine. |
Dive into Deep Learning for Natural Language Processing (D19-2)
Copied to clipboard
| Challenge: | GluonNLP is a powerful new toolkit that automates the most laborious aspects of deep learning for NLP. |
| Approach: | This hands-on tutorial demonstrates how to scale unsupervised pre-training techniques with Apache MXNet and GluonNLP. |
| Outcome: | This hands-on tutorial examines the challenges of scaling these models and algorithms effectively with Apache MXNet and GluonNLP. |
DEM: Distribution Edited Model for Training with Mixed Data Distributions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent fine-tuning approaches for large language models require supervised finetun on diverse datasets and follow different distributions. |
| Approach: | They propose a distribution edited model that integrates models individually trained on each data source with the base model using basic element-wise vector operations. |
| Outcome: | The proposed model outperforms baseline models on a variety of benchmarks and is cheaper than standard data mixing methods. |
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained transformer models have demonstrated remarkable performance across various natural language processing tasks. |
| Approach: | They propose a transformer variant with mixed attention spans that leverages the attention mechanism to capture long- and short-range dependencies in the sequence. |
| Outcome: | The proposed model can achieve competitive performance to models with full attention while reducing computational cost (75%) |
Sequence-level Large Language Model Training with Contrastive Preference Optimization (2025.findings-naacl)
Copied to clipboard
| Challenge: | a new method to improve the performance of large language models requires a small computational cost. |
| Approach: | They propose a CPO procedure that can inject sequence-level information into the model at any training stage without expensive human labeled data. |
| Outcome: | The proposed objective surpasses the next token prediction in terms of win rate in instruction-following and text generation tasks. |
Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual (D19-61)
Copied to clipboard
| Challenge: | Statistical natural language inference models are susceptible to learning dataset bias. |
| Approach: | They propose a debiasing algorithm that debiases models that use only known dataset biases . they use two benchmark datasets to train three high-performing NLI models . |
| Outcome: | The proposed learning objective improves model performance on challenge datasets while maintaining reasonable performance on original datasets. |