Papers by Santiago Ontanon
FNet: Mixing Tokens with Fourier Transforms (2022.naacl-main)
Copied to clipboard
| Challenge: | Using simple linear transformations, Transformer encoders can be sped up with limited accuracy costs by replacing the self-attention sublayers with simple linear mixing mechanisms. |
| Approach: | They propose to replace the self-attention sublayer with a linear transformation that "mixes" input tokens. |
| Outcome: | The proposed model outperforms the “efficient Transformers” on the GLUE benchmark at longer input lengths and on smaller models with a light memory footprint. |
LongT5: Efficient Text-To-Text Transformer for Long Sequences (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has shown that increasing the input length or increasing model size can improve the performance of Transformer-based neural models. |
| Approach: | They propose a model that integrates attention ideas from long-input transformers and adopts pre-training strategies from summarization pre-train into the scalable T5 architecture. |
| Outcome: | The proposed model outperforms the original T5 models on several summarization and question answering tasks and achieves state-of-the-art results. |
Improving Compositional Generalization in Classification Tasks via Structure Annotations (2021.acl-short)
Copied to clipboard
| Challenge: | Compositional generalization is the ability to generalize systematically to a new data distribution by combining known components. |
| Approach: | They propose to convert a natural language sequence-to-sequence dataset into a classification dataset that requires compositional generalization. |
| Outcome: | The proposed model can generalize compositionally by providing hints on the structure of the input. |
ETC: Encoding Long and Structured Inputs in Transformers (2020.emnlp-main)
Copied to clipboard
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, Li Yang
| Challenge: | Existing models for natural language processing (NLP) have been challenging to scale attention to longer inputs. |
| Approach: | They propose an extended Transformer construction architecture that scales attention to longer inputs by combining global-local attention with relative position encodings and a "Contrastive Predictive Coding" objective. |
| Outcome: | The proposed architecture scales attention to longer inputs and encodes structured inputs. |
Making Transformers Solve Compositional Tasks (2022.acl-long)
Copied to clipboard
| Challenge: | Several studies have reported the inability of Transformer models to generalize compositionally . a key aspect of natural language is the ability to learn basic primitives . |
| Approach: | They propose to use Transformers to generalize compositionally in a large range of tasks . they find that Transformers generalize significantly better than previous models . |
| Outcome: | The proposed models generalize compositionally significantly better than previous models . a set of 12 datasets shows that the proposed models can be improved . |
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new text-to-text transformer is suitable for multilingual inputs . many of the current models are English-only, making them inapplicable to other languages. |
| Approach: | They propose to extend a multilingual text-to-text transformer to handle long inputs . they use the mC4 dataset to pretrain the model to handle multilingual data . |
| Outcome: | The proposed model performs well on multilingual summarization and question-answering tasks. |
MEMORY-VQ: Compression for Tractable Internet-Scale Memory (2024.naacl-short)
Copied to clipboard
Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis, Santiago Ontanon, William Cohen, Sumit Sanghai, Joshua Ainslie
| Challenge: | Memory-based methods like LUMEN pre-compute token representations for retrieved passages to speed up inference. |
| Approach: | They propose a method to reduce storage requirements of memory-augmented models . they use a vector quantization variational autoencoder to compress token representations . |
| Outcome: | The proposed method achieves 16x compression rate with comparable performance on KILT benchmark. |
CoLT5: Faster Long-Range Transformers with Conditional Computation (2023.emnlp-main)
Copied to clipboard
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontanon, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai
| Challenge: | Many natural language processing tasks require long inputs, but processing long documents with a Transformer model is expensive due to quadratic attention complexity and applying feedforward and attention projection layers to every input token. |
| Approach: | They propose a long-input Transformer model that builds on the intuition that some tokens are more important than others and uses conditional computation to devote more computation to important tokens. |
| Outcome: | The proposed model achieves stronger performance than LongT5 with faster training and inference, achieving SOTA on the long-input SCROLLS benchmark. |