Papers by Jason Liu
Language Model Augmented Relevance Score (2021.acl-long)
Copied to clipboard
| Challenge: | Existing metrics that compare the candidate with the human reference do not consider the context, resulting in poor correlation with human judgements. |
| Approach: | They propose a language model-aware metric that augments the human reference while considering the context to provide evaluation scores that correlate highly with human judgements. |
| Outcome: | The proposed metric achieves higher correlation with human reference judgements and differentiates well-formed candidates from adversarial samples to a larger degree. |
Comparing Test Sets with Item Response Theory (2021.acl-long)
Copied to clipboard
Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman
| Challenge: | Recent results from large pretrained models show that many datasets are saturated and unlikely to detect further progress. |
| Approach: | They evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. |
| Outcome: | The proposed datasets are saturated and unlikely to detect future improvements. |
Recipes for Building an Open-Domain Chatbot (2021.eacl-main)
Copied to clipboard
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, Jason Weston
| Challenge: | Existing work shows that scaling models in the number of parameters and the size of the data they are trained on gives improved results, but other factors are important. |
| Approach: | They propose to build open-domain chatbots that can be scaled to improve their performance . they use a blend of cognitive and cognitive skills to build a model that combines these skills . |
| Outcome: | The proposed models outperform existing approaches in multi-turn dialogue on engagingness and humanness measurements. |
The Linearity of the Effect of Surprisal on Reading Times across Languages (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large amount of insight into human language processing can be gleaned by studying word-by-word processing difficulty. |
| Approach: | They extend the study by examining eyetracking corpora of seven languages . they find evidence for superlinearity in some languages, but highly sensitive to language models . |
| Outcome: | The study extends existing studies on english to Danish, Dutch, English, German, Japanese, Mandarin, and Russian. |
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020.acl-main)
Copied to clipboard
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman
| Challenge: | Unsupervised pretraining has recently pushed the state of the art on many natural language understanding tasks. |
| Approach: | They perform a large-scale survey on a pretrained RoBERTa model with 110 intermediate-target task combinations and 25 probing tasks to reveal the specific skills that drive transfer. |
| Outcome: | The proposed model is trained on 110 intermediate-target task combinations and compared with 25 probing tasks to reveal the specific skills that drive transfer. |
Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs (D19-1)
Copied to clipboard
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, Samuel R. Bowman
| Challenge: | Recent work evaluating sentence representation models' knowledge of grammar has been slower to emerge. |
| Approach: | They propose five experimental methods inspired by prior work evaluating pretrained sentence representation models to examine their grammatical knowledge. |
| Outcome: | The proposed methods show that the model has significant knowledge of the licensing environment but its success varies widely across different methods. |
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models (2020.acl-demos)
Copied to clipboard
Yada Pruksachatkun, Phil Yeres, Haokun Liu, Jason Phang, Phu Mon Htut, Alex Wang, Ian Tenney, Samuel R. Bowman
| Challenge: | jiant is an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. |
| Approach: | They introduce jiant, an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. |
| Outcome: | The proposed toolkit reproduces published performance on GLUE and SuperGLUE tasks. |
Automated Generation of Accurate & Fluent Medical X-ray Reports (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing medical report generation efforts focus on producing human-readable reports, yet the generated text may not be well aligned to the clinical facts. |
| Approach: | They propose to automate the generation of medical reports from chest X-ray image inputs . medical reports are the primary medium, which physicians communicate findings from scans - authors say . |
| Outcome: | The proposed method achieves fluency and clinical accuracy on common metrics. |
Sandcastles in the Storm: Revisiting the (Im)possibility of Strong Watermarking (2025.acl-long)
Copied to clipboard
Fabrice Y Harel-Canada, Boran Erol, Connor Choi, Jason Liu, Gary Jiarui Song, Nanyun Peng, Amit Sahai
| Challenge: | Recent theoretical work argues that any watermark can be erased via random walk attacks that perturb text while preserving quality. |
| Approach: | They propose that any watermark can be erased via random walk attacks that perturb text while preserving quality. |
| Outcome: | The proposed method underperforms the theoretical models in large-scale experiments and human-validated assessments. |
SHIELD: LLM-Driven Schema Induction for Predictive Analytics in EV Battery Supply Chain Disruptions (2024.emnlp-industry)
Copied to clipboard
Zhi-Qi Cheng, Yifei Dong, Aike Shi, Wei Liu, Yuzhi Hu, Jason O’Connor, Alexander Hauptmann, Kate Whitefoot
| Challenge: | EV battery supply chain is vulnerable to disruptions caused by natural disasters and geopolitical tensions. |
| Approach: | They propose a system integrating Large Language Models with domain expertise for EV supply chain risk assessment. |
| Outcome: | Evaluated on 12,070 paragraphs from 365 sources (2022-2023), SHIELD outperforms baseline GCNs and LLM+prompt methods in disruption prediction. |
Modulating Language Models with Emotions (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generating context-aware language that embodies diverse emotions are dull or generic due to limited training data for diverse emotions. |
| Approach: | They propose a modulated layer normalization technique that generates emotional responses using large pre-trained models. |
| Outcome: | The proposed method outperforms baseline methods on the MojiTalk dataset while maintaining diversity, fluency, and coherence. |
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too (2020.aacl-main)
Copied to clipboard
Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Samuel R. Bowman
| Challenge: | a study of intermediate-task training in monolingual English shows that it improves model performance on non-English language understanding tasks. |
| Approach: | They evaluate whether English intermediate-task training is still helpful on non-English target tasks . BUCC and Tatoeba sentence retrieval tasks see large improvements . |
| Outcome: | The proposed model outperforms existing models on non-English language understanding tasks. |
Investigating Efficiently Extending Transformers for Long Input Summarization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained Transformer models have proven capable at tackling natural language tasks, but handling long sequence inputs still poses a significant challenge. |
| Approach: | They propose an extension of the PEGASUS model with additional long input pretraining to handle inputs of up to 16K tokens. |
| Outcome: | The proposed model achieves strong performance on long input summarization tasks comparable with much larger models. |
LLMs for Bayesian Optimization in Scientific Domains: Are We There Yet? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have been proposed as general-purpose agents for experimental design . eval: LLMs show no sensitivity to experimental feedback. |
| Approach: | They propose a method that combines LLM prior knowledge with nearest-neighbor sampling to guide the design of experiments. |
| Outcome: | The proposed method outperforms classical methods in the design of experiments. |
A Critical Analysis of Document Out-of-Distribution Detection (2023.findings-emnlp)
Copied to clipboard
Jiuxiang Gu, Yifei Ming, Yi Zhou, Jason Kuen, Vlad Morariu, Handong Zhao, Ruiyi Zhang, Nikolaos Barmpalios, Anqi Liu, Yixuan Li, Tong Sun, Ani Nenkova
| Challenge: | Existing document understanding models focus on single-modal inputs such as images or texts. |
| Approach: | They propose to use a spatial-aware adapter to adapt transformer-based language models to document domain to exploit multi-modal information. |
| Outcome: | The proposed model significantly improves the OOD detection performance compared to using a standard language model and to competitive baselines. |