Papers by Matt Smith
Linguistic Knowledge and Transferability of Contextual Representations (N19-1)
Copied to clipboard
| Challenge: | Recent work has explored contextual word representations, which assign each word a vector that is a function of the entire input sequence. |
| Approach: | They compare pretrained word representations with 16 diverse probing tasks to examine their transferability. |
| Outcome: | The pretrained representations are successful across a diverse set of NLP tasks . the models are competitive with state-of-the-art models but fail on fine-grained tasks requiring fine-granular knowledge, the study finds . |
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning (D19-1)
Copied to clipboard
| Challenge: | Existing reading comprehension benchmarks do not contain complex coreferential phenomena . obtaining questions focused on such phenomena is difficult because of lexical cues . |
| Approach: | They propose to use a crowdsourced dataset to examine the ability of models to resolve coreference among entities in Wikipedia paragraphs. |
| Outcome: | The proposed model performs significantly worse than humans on the reading comprehension benchmark . paragraphs and other longer texts typically make multiple references to the same entities . |
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model (2024.emnlp-industry)
Copied to clipboard
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, Kavya Srinet, Babak Damavandi, Anuj Kumar
| Challenge: | Prior work on LLMs focused on models that combine text and one other modality, such as image encoders or proprietary models that are not open sourced. |
| Approach: | They propose a unified model that reasons over diverse input modality signals and generates textual responses. |
| Outcome: | The proposed model performs better on multimodal tasks than industry-leading models . |
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020.findings-emnlp)
Copied to clipboard
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou
| Challenge: | Standard test sets for supervised learning evaluate in-distribution generalization but are misleading when a dataset has systematic gaps. |
| Approach: | They propose a more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data. |
| Outcome: | The proposed model performs significantly lower on contrast sets than on the original test sets—up to 25% in some cases. |
Competency Problems: On Finding and Removing Artifacts in Language Data (2021.emnlp-main)
Copied to clipboard
Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, Noah A. Smith
| Challenge: | Recent work in NLP has documented dataset artifacts, bias, and spurious correlations . how to tell which features have spurious instead of legitimate correlations is typically left unspecified . |
| Approach: | They propose a class of competency problems to formalize this notion into a classification . they show that realistic datasets will increasingly deviate from competency problems . |
| Outcome: | The proposed model can be used to show that models are inappropriately affected by these less extreme biases. |
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing information-seeking question answering datasets do not perform well on answering these questions . existing models that do well on other QA tasks do not do well answering these tasks . |
| Approach: | They present a dataset of 5049 questions over 1585 NLP papers . they use a question-seeking QA model that seeks information in the full text . |
| Outcome: | The proposed dataset underperforms existing models on other QA tasks by 27 F1 points . the focus is on document-grounded, information-seeking QA . |