Papers by Yada Pruksachatkun
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020.acl-main)
Copied to clipboard
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman
| Challenge: | Unsupervised pretraining has recently pushed the state of the art on many natural language understanding tasks. |
| Approach: | They perform a large-scale survey on a pretrained RoBERTa model with 110 intermediate-target task combinations and 25 probing tasks to reveal the specific skills that drive transfer. |
| Outcome: | The proposed model is trained on 110 intermediate-target task combinations and compared with 25 probing tasks to reveal the specific skills that drive transfer. |
On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations (2022.acl-short)
Copied to clipboard
Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, Aram Galstyan
| Challenge: | Recent natural language processing systems use large language models as the backbone . however, societal biases are encoded in these models and transferred to downstream applications . |
| Approach: | They propose to use two categories to measure fairness in natural language processing tasks . they find intrinsic and extrinsic metrics do not correlate in their original setting . |
| Outcome: | The proposed metrics do not correlate in their original setting, the authors show . they find that they are not accurate when correcting for metric misalignments and noise . |
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models (2020.acl-demos)
Copied to clipboard
Yada Pruksachatkun, Phil Yeres, Haokun Liu, Jason Phang, Phu Mon Htut, Alex Wang, Ian Tenney, Samuel R. Bowman
| Challenge: | jiant is an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. |
| Approach: | They introduce jiant, an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. |
| Outcome: | The proposed toolkit reproduces published performance on GLUE and SuperGLUE tasks. |
Leveraging Explicit Procedural Instructions for Data-Efficient Action Prediction (2023.findings-acl)
Copied to clipboard
| Challenge: | a paper presents a data-efficient solution to constructing task-oriented dialogue systems . large language models have shown success in modeling such dialogues, but they require large quantities of data . |
| Approach: | They propose a system that leverages explicit instructions from agent guidelines . they propose dialogue-document matching and action-oriented masked language modeling . |
| Outcome: | The proposed system improves accuracy predicting in- and out-of-distribution actions while preserving high performance in settings with low or sparse data. |
CLIP: A Dataset for Extracting Action Items for Physicians from Hospital Discharge Notes (2021.acl-long)
Copied to clipboard
James Mullenbach, Yada Pruksachatkun, Sean Adler, Jennifer Seale, Jordan Swartz, Greg McKelvey, Hui Dai, Yi Yang, David Sontag
| Challenge: | Continuity of care is crucial to ensuring positive health outcomes for patients discharged from an inpatient hospital setting. |
| Approach: | They propose to annotate clinical action items from a dataset of medical notes annotated by physicians and extract them as multi-aspect extractive summarization. |
| Outcome: | The proposed dataset is annotated by physicians and covers 718 documents representing 100K sentences. |
Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to reduce disparities in model outcomes have focused on data augmentation, debiasing model embeddings, or adding fairness-based optimization objectives during training. |
| Approach: | They propose to use certified word substitution robustness methods to improve equality of odds and equality of opportunity on multiple text classification tasks. |
| Outcome: | The proposed methods improve equality of odds and equality of opportunity on multiple text classification tasks. |
Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal (2022.findings-acl)
Copied to clipboard
Umang Gupta, Jwala Dhamala, Varun Kumar, Apurv Verma, Yada Pruksachatkun, Satyapriya Krishna, Rahul Gupta, Kai-Wei Chang, Greg Ver Steeg, Aram Galstyan
| Challenge: | Language models excel at generating coherent text, but can be biased in multiple ways, including the unfounded association of male and female genders with gender-neutral professions. |
| Approach: | They propose to modify teacher probabilities and augment the training set to learn a fair model during knowledge distillation by modifying teacher probability and augmenting the training sets. |
| Outcome: | The proposed approach reduces gender disparity in open-ended text generated from the distilled and finetuned models with only a minor compromise in utility. |
Measuring Fairness of Text Classifiers via Prediction Sensitivity (2022.acl-long)
Copied to clipboard
| Challenge: | Existing fairness metrics are not yet available to measure the fairness of language processing systems. |
| Approach: | They propose a new metric which measures fairness in machine learning models based on the model’s prediction sensitivity to perturbations in input features. |
| Outcome: | The proposed metric can be linked with a specific notion of group fairness and individual fairness, and correlates well with humans’ perception of fairness. |
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too (2020.aacl-main)
Copied to clipboard
Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Samuel R. Bowman
| Challenge: | a study of intermediate-task training in monolingual English shows that it improves model performance on non-English language understanding tasks. |
| Approach: | They evaluate whether English intermediate-task training is still helpful on non-English target tasks . BUCC and Tatoeba sentence retrieval tasks see large improvements . |
| Outcome: | The proposed model outperforms existing models on non-English language understanding tasks. |