Papers by Mandy Guo
Neural Retrieval for Question Answering with Cross-Attention Supervised Data Augmentation (2021.acl-short)
Copied to clipboard
| Challenge: | Early fusion models with cross-attention have shown better-than-human performance on some question answer benchmarks, while it is a poor fit for retrieval since it prevents pre-computation of the answer representations. |
| Approach: | They propose a supervised data mining method to train an efficient late fusion retrieval model by using cross-attention models with cross-references. |
| Outcome: | The proposed model outperforms retrieval models trained with gold annotations on Precision at N (P@N) and Mean Reciprocal Rank (MRR). |
Multilingual Universal Sentence Encoder for Semantic Retrieval (2020.acl-demos)
Copied to clipboard
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, Ray Kurzweil
| Challenge: | Using a multi-task trained dual-encoder, our models embed text from 16 languages into a shared semantic space. |
| Approach: | They propose retrieval focused multilingual sentence embedding models on TensorFlow Hub. |
| Outcome: | The models achieve state-of-the-art on monolingual and cross-lingual retrieval (SR) and retrieval question answering (ReQA) competitive performance is obtained on related tasks of translation pair bitext retrieval and retrieving question answering. |
LongT5: Efficient Text-To-Text Transformer for Long Sequences (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has shown that increasing the input length or increasing model size can improve the performance of Transformer-based neural models. |
| Approach: | They propose a model that integrates attention ideas from long-input transformers and adopts pre-training strategies from summarization pre-train into the scalable T5 architecture. |
| Outcome: | The proposed model outperforms the original T5 models on several summarization and question answering tasks and achieves state-of-the-art results. |
TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text style transfer require style-labeled training data, but use only labeled data at inference time. |
| Approach: | They propose a method that uses readily-available unlabeled text to train style transfer . they use a style vector to condition a decoder to perform style transfer using unlabelled text . |
| Outcome: | The proposed method is competitive on sentiment transfer, even compared to models trained fully on labeled data. |
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding (2023.emnlp-main)
Copied to clipboard
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan Plummer, Kate Saenko, Jianmo Ni, Mandy Guo
| Challenge: | Existing datasets for webpages contain only fragments of webpages . generative tasks like page description generation and section summarization are often left unstudied . |
| Approach: | They introduce a Wikipedia Webpage suite that contains 2M pages with all associated image, text, and structure data. |
| Outcome: | The proposed approach performs better than full attention with lower computational complexity. |
MURAL: Multimodal, Multitask Representations Across Languages (2021.findings-emnlp)
Copied to clipboard
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, Jason Baldridge
| Challenge: | Image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages. |
| Approach: | They propose a dual encoder that integrates image-text matching and translation pairs to solve two tasks by learning from billions of pairs. |
| Outcome: | The proposed encoder outperforms ALIGN's cross-modal retrieval performance on well-resourced languages and significantly improves on under-resource languages. |
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new text-to-text transformer is suitable for multilingual inputs . many of the current models are English-only, making them inapplicable to other languages. |
| Approach: | They propose to extend a multilingual text-to-text transformer to handle long inputs . they use the mC4 dataset to pretrain the model to handle multilingual data . |
| Outcome: | The proposed model performs well on multilingual summarization and question-answering tasks. |
Wiki-40B: Multilingual Language Model Dataset (2020.lrec-1)
Copied to clipboard
| Challenge: | We propose a new multilingual language model benchmark that is composed of 40+ languages spanning several scripts and linguistic families. |
| Approach: | They propose a multilingual language model benchmark composed of 40+ languages . they train monolingual causal language models using a state-of-the-art model . |
| Outcome: | The proposed model is composed of 40+ languages spanning several scripts and linguistic families. |
CoLT5: Faster Long-Range Transformers with Conditional Computation (2023.emnlp-main)
Copied to clipboard
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontanon, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai
| Challenge: | Many natural language processing tasks require long inputs, but processing long documents with a Transformer model is expensive due to quadratic attention complexity and applying feedforward and attention projection layers to every input token. |
| Approach: | They propose a long-input Transformer model that builds on the intuition that some tokens are more important than others and uses conditional computation to devote more computation to important tokens. |
| Outcome: | The proposed model achieves stronger performance than LongT5 with faster training and inference, achieving SOTA on the long-input SCROLLS benchmark. |