Papers by Vinh Tran
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2023.findings-emnlp)
Copied to clipboard
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, Donald Metzler
| Challenge: | Existing studies on the scaling properties of model architectures have not explored the impact of inductive biases on scaling behaviour. |
| Approach: | They conduct extensive experiments to understand scaling behaviour of ten different model architectures. |
| Outcome: | The results show that the best performing model can fluctuate at different scales. |
DSI++: Updating Transformer Memory with New Documents (2023.emnlp-main)
Copied to clipboard
Sanket Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Tran, Jinfeng Rao, Marc Najork, Emma Strubell, Donald Metzler
| Challenge: | Differentiable Search Indices (DSIs) encode a corpus of documents and use the same model to map queries directly to relevant document identifiers. |
| Approach: | They propose a continual learning challenge for Differentiable Search Indices (DSIs) they propose to continuously index new documents while answering queries related to previously and newly indexed documents. |
| Outcome: | The proposed model stably memorizes more documents and improves the average Hits@10 by +21.1% over baselines. |
Transcending Scaling Laws with 0.1% Extra Compute (2023.emnlp-main)
Copied to clipboard
Yi Tay, Jason Wei, Hyung Chung, Vinh Tran, David So, Siamak Shakeri, Xavier Garcia, Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc Le, Mostafa Dehghani
| Challenge: | Existing scaling of language models is expensive and requires significant computational costs. |
| Approach: | They propose a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute. |
| Outcome: | The proposed method significantly improves existing language models and their scaling curves with a relatively tiny amount of extra compute. |
How Does Generative Retrieval Scale to Millions of Passages? (2023.emnlp-main)
Copied to clipboard
Ronak Pradeep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, Vinh Tran
| Challenge: | generative retrieval is a new paradigm for information retrieval, enabling a sequence-to-sequence model with a single Transformer . generative encoders have been used on small corpora, but only on large ones . |
| Approach: | They propose to encode an entire document corpus within a single Transformer . they find generative retrieval is competitive with state-of-the-art dual encoders on small corpora . |
| Outcome: | The proposed approach is competitive with state-of-the-art dual encoders on small corpora, the study finds . the proposed approach only evaluates on document corporales on the order of 100K in size . |
Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification (2022.emnlp-industry)
Copied to clipboard
Jai Gupta, Yi Tay, Chaitanya Kamath, Vinh Tran, Donald Metzler, Shailesh Bavadekar, Mimi Sun, Evgeniy Gabrilovich
| Challenge: | a new method for classification of COVID-19 vaccination related search queries is proposed . the proposed method uses pretrained Transformers and dense features to generate search insights . |
| Approach: | They propose a machine learning model that detects COVID-19 vaccination related search queries . they use pretrained Transformers to consider dense features as memory tokens that the model can attend to . |
| Outcome: | The proposed model improves the Vaccine Search Insights task by +15% . the proposed model uses pretrained Transformers and traditional dense features . |
AgreeSum: Agreement-Oriented Multi-Document Summarization (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on agreement-oriented multidocument summarization have focused on clusters of articles . a recent study focused on the use of a pretraining framework to summarize articles based on the "union" of the articles. |
| Approach: | They propose to use agreement-oriented multidocument summarization to provide agreement-orientated summaries that represent information common to all articles. |
| Outcome: | The proposed task is called agreement-oriented multidocument summarization . the authors apply the pretrained model PEGASUS onto the task . |