Papers by Matthew Finlayson
What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on instruction learning, where a model learns to perform unseen tasks from task descriptions alone. |
| Approach: | They propose to use a controlled synthetic environment to characterize large transformer models as instruction learners. |
| Outcome: | The proposed model can interpret only 65.6% of test instructions and 11%-24% of instructions in out-of-distribution settings. |
Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts. |
| Approach: | They apply causal mediation analysis to pre-trained neural language models to investigate their models' preferences for grammatical inflections and whether neurons process subject-verb agreement similarly across sentences with different syntactic structures. |
| Outcome: | The proposed model can predict correct token from grammatically minimally different continuations with high accuracy even in difficult contexts. |
LILA: A Unified Benchmark for Mathematical Reasoning (2022.emnlp-main)
Copied to clipboard
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, Ashwin Kalyan
| Challenge: | Towards evaluating and improving AI systems in this domain, we propose a mathematical reasoning benchmark based on 23 diversetasks . |
| Approach: | They propose a mathematical reasoning benchmark that includes 23 diverse tasks . they extend the benchmark by collecting task instructions and solutions in the form of Python programs . |
| Outcome: | The proposed model improves on multi-tasking while the best performing model only achieves 60.40%. |
Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracy (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are used to discriminate on multiple-choice tasks that place probability mass on vocabulary tokens that aren’t among the given answer choices. |
| Approach: | They propose a mathematical formalism for SFC which allows us to quantify and bound its impact for the first time. |
| Outcome: | The proposed method eliminates the impact of SFC in the majority of instances. |