Papers by Yaoliang Yu
The Art of Abstention: Selective Prediction and Error Regularization for Natural Language Processing (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have improved the state-of-the-art results on many NLP applications. |
| Approach: | They propose a simple error regularization trick that improves confidence estimation without substantially increasing the computation budget. |
| Outcome: | The proposed regularization improves confidence estimation without increasing computation budget. |
Operator Selection and Ordering in a Pipeline Approach to Efficiency Optimizations for Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | Natural language processing tasks rely on complex neural models . transformer-based models are typically slow to execute, making it a non-trivial challenge to apply them in real-world applications. |
| Approach: | They propose to consider an efficiency method as an operator applied on a model . they find that the commutativity and cumulativeness of efficiency operators are plausible . |
| Outcome: | The proposed method is commutative and cumulative, and the results are estimated by combining methods. |
What Part of the Neural Network Does This? Understanding LSTMs by Measuring and Dissecting Neurons (D19-1)
Copied to clipboard
| Challenge: | Biological neural systems consist of a huge number of neurons, and can react to the environment in complicated ways. |
| Approach: | They propose a metric to quantify the sensitivity of neurons to each label and conduct experiments to prove it. |
| Outcome: | The proposed metric is based on a set of experiments that show that dropping an arbitrary neuron significantly degrades the accuracy of the model. |
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods to make exiting decisions are limited to classification tasks . large-scale pre-trained language models such as BERT have brought performance gain but at the cost of heavy computational burden. |
| Approach: | They propose a fine-tuning strategy and a learning-to-exit module to accelerate BERT inference . they propose to make trade-offs between model quality and efficiency by early exiting . |
| Outcome: | The proposed approach improves early exiting for BERT, with better trade-offs . it can be combined with other acceleration methods, and the proposed strategy can be applied to regression tasks. |
Showing Your Work Doesn’t Always Work (2020.acl-main)
Copied to clipboard
| Challenge: | countless experimental papers lack empirical rigor, disregarding necessities such as statistical significance tests and computational environments. |
| Approach: | They propose to report the expected validation effectiveness of the best-tuned model with respect to the computational budget. |
| Outcome: | The proposed model favors negative errors and yields poor bootstrapped confidence intervals, the authors argue . they find that the proposed model is biased and uses error-prone assumptions . |
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models such as BERT are notorious for being slow in both training and inference. |
| Approach: | They propose a method to accelerate BERT inference by inserting extra classification layers between each transformer layer of BERT. |
| Outcome: | The proposed method saves up to 40% inference time with minimal degradation in model quality. |
Posterior Differential Regularization with f-divergence for Improving Model Robustness (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained models suffer catastrophic degradation in out-of-domain generalization to datasets with domain shift or adversarial scenarios. |
| Approach: | They propose to regularize the posterior difference between clean and noisy inputs by using a Jacobian regularization framework and a virtual adversarial training framework. |
| Outcome: | The proposed framework can improve model robustness in fully supervised and semi-supervised settings. |