Papers by Donald Metzler
LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction (2023.acl-long)
Copied to clipboard
Jeremiah Milbauer, Annie Louis, Mohammad Javad Hosseini, Alex Fabrikant, Donald Metzler, Tal Schuster
| Challenge: | In many NLP tasks, the input text can be seen as a sequence of related segments. |
| Approach: | They propose a layer-adjustable interactions framework that contextualizes token representations by attending to all other tokens at each layer, leading to quadratic increase in compute effort with the input length. |
| Outcome: | The proposed model reduces 30-50% of attention FLOPs while maintaining high accuracy. |
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2023.findings-emnlp)
Copied to clipboard
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, Donald Metzler
| Challenge: | Existing studies on the scaling properties of model architectures have not explored the impact of inductive biases on scaling behaviour. |
| Approach: | They conduct extensive experiments to understand scaling behaviour of ten different model architectures. |
| Outcome: | The results show that the best performing model can fluctuate at different scales. |
StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models that induce grammar structures from data focus on constituency or dependency structures alone. |
| Approach: | They propose a model that can induce dependency and constituency structure at the same time. |
| Outcome: | The proposed model can induce both constituency and dependency structures at the same time. |
DSI++: Updating Transformer Memory with New Documents (2023.emnlp-main)
Copied to clipboard
Sanket Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Tran, Jinfeng Rao, Marc Najork, Emma Strubell, Donald Metzler
| Challenge: | Differentiable Search Indices (DSIs) encode a corpus of documents and use the same model to map queries directly to relevant document identifiers. |
| Approach: | They propose a continual learning challenge for Differentiable Search Indices (DSIs) they propose to continuously index new documents while answering queries related to previously and newly indexed documents. |
| Outcome: | The proposed model stably memorizes more documents and improves the average Hits@10 by +21.1% over baselines. |
Transcending Scaling Laws with 0.1% Extra Compute (2023.emnlp-main)
Copied to clipboard
Yi Tay, Jason Wei, Hyung Chung, Vinh Tran, David So, Siamak Shakeri, Xavier Garcia, Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc Le, Mostafa Dehghani
| Challenge: | Existing scaling of language models is expensive and requires significant computational costs. |
| Approach: | They propose a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute. |
| Outcome: | The proposed method significantly improves existing language models and their scaling curves with a relatively tiny amount of extra compute. |
How Reliable are Model Diagnostics? (2021.findings-acl)
Copied to clipboard
| Challenge: | Contemporary statistical models trade off interpretability and simplicity for powerful parameterizations and inductive biases, enabling impressive performance. |
| Approach: | They examine three recent models and find they are not yet reliable . they also formulate recommendations for practitioners and researchers . |
| Outcome: | The proposed models are not as reliable as previously assumed, the authors argue . their findings suggest that they are needed for improving models and training setups . |
Are Pretrained Convolutions Better than Pretrained Transformers? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent research has shown promise in entirely convolutional, or CNN, architectures, but they have not been explored using the pre-train-fine-tune paradigm. |
| Approach: | They propose to use the pre-train-fine-tune paradigm to study convolutional models. |
| Outcome: | The proposed architectures outperform Transformers in certain scenarios, but with caveats. |
How Does Generative Retrieval Scale to Millions of Passages? (2023.emnlp-main)
Copied to clipboard
Ronak Pradeep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, Vinh Tran
| Challenge: | generative retrieval is a new paradigm for information retrieval, enabling a sequence-to-sequence model with a single Transformer . generative encoders have been used on small corpora, but only on large ones . |
| Approach: | They propose to encode an entire document corpus within a single Transformer . they find generative retrieval is competitive with state-of-the-art dual encoders on small corpora . |
| Outcome: | The proposed approach is competitive with state-of-the-art dual encoders on small corpora, the study finds . the proposed approach only evaluates on document corporales on the order of 100K in size . |
Reverse Engineering Configurations of Neural Text Generation Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent advances in neural text generation modeling have raised concerns about how such approaches might be used in malicious ways. |
| Approach: | They propose to distinguish which of several variants of a given model generated some piece of text by performing diagnostic tests. |
| Outcome: | The proposed method identifies which of several variants of a given model generated some piece of text and if so, if it is more sensitive to different modeling choices than previously thought. |
SEMQA: Semi-Extractive Multi-Source Question Answering (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent proposed long-form question answering systems have shown promising capabilities, but attributing and verifying their generated abstractive answers can be difficult. |
| Approach: | They propose a task that summarises multiple sources in a semi-extractive fashion . they create a dataset with human-written semi-extractive answers to natural and generated questions . |
| Outcome: | The proposed task summarizes multiple sources in a semi-extractive fashion and produces fine in-line attributions by-design that are easy to verify, interpret, and evaluate. |
OpenMSD: Towards Multilingual Scientific Documents Similarity Measurement (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for finding related papers in different languages are not effective for multilingual SDSM. |
| Approach: | They propose to use Open-access Multilingual Scientific Documents to develop multilingual SDSM models that adjust and extend state-of-the-art methods for English SDSM tasks. |
| Outcome: | The proposed model outperforms baseline methods on multilingual SDSM tasks while preserving the performance of the existing methods. |
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2024.findings-naacl)
Copied to clipboard
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, Michael Bendersky
| Challenge: | Existing methods to rank documents using large language models do not understand these challenging ranking formulations. |
| Approach: | They propose to use Pairwise Ranking Prompting to improve ranking performance . they propose to outperform fine-tuned baseline rankers on benchmark datasets . |
| Outcome: | The proposed technique outperforms supervised baselines on benchmark datasets and outperformed other LLM-based solutions by over 10% on average. |
PaRaDe: Passage Ranking using Demonstrations with LLMs (2023.findings-emnlp)
Copied to clipboard
Andrew Drozdov, Honglei Zhuang, Zhuyun Dai, Zhen Qin, Razieh Rahimi, Xuanhui Wang, Dana Alon, Mohit Iyyer, Andrew McCallum, Donald Metzler, Kai Hui
| Challenge: | Existing studies show that large language models can be instructed to perform zero-shot passage re-ranking . Existing work like UPR demonstrate promising results for zero- shot ranking using LLMs . |
| Approach: | They propose a demonstration selection strategy based on difficulty rather than semantic similarity . they propose to include only one demonstration in the prompt to improve re-ranking . |
| Outcome: | The proposed method improves LLM-based re-ranking by adding one demonstration to the prompt. |
Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification (2022.emnlp-industry)
Copied to clipboard
Jai Gupta, Yi Tay, Chaitanya Kamath, Vinh Tran, Donald Metzler, Shailesh Bavadekar, Mimi Sun, Evgeniy Gabrilovich
| Challenge: | a new method for classification of COVID-19 vaccination related search queries is proposed . the proposed method uses pretrained Transformers and dense features to generate search insights . |
| Approach: | They propose a machine learning model that detects COVID-19 vaccination related search queries . they use pretrained Transformers to consider dense features as memory tokens that the model can attend to . |
| Outcome: | The proposed model improves the Vaccine Search Insights task by +15% . the proposed model uses pretrained Transformers and traditional dense features . |
ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference (2022.findings-acl)
Copied to clipboard
Kai Hui, Honglei Zhuang, Tao Chen, Zhen Qin, Jing Lu, Dara Bahri, Ji Ma, Jai Gupta, Cicero Nogueira dos Santos, Yi Tay, Donald Metzler
| Challenge: | State-of-the-art neural models typically encode document-query pairs using cross-attention for re-ranking. |
| Approach: | They propose to fine tune a pretrained encoder-decoder model using document to query generation. |
| Outcome: | The proposed model achieves comparable results to more expensive approaches while being 6.8X faster. |
Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in modeling and datasets demonstrate promising performance for NLI. |
| Approach: | They explore the direct zero-shot applicability of NLI models to real applications . they analyze the robustness of models to longer and out-of-domain inputs . |
| Outcome: | The proposed models are robust to longer and out-of-domain inputs and can perform on full documents. |
Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts? (2025.findings-naacl)
Copied to clipboard
| Challenge: | a recent study shows that human understanding of text depends on general semantic concepts of words that are robust to their superficial forms. |
| Approach: | They evaluate the accuracy of multilingual multilingual language models based on subword-level semantics . they form "semantic tokens" by merging semantically similar subwords and embeddings based upon the results . |
| Outcome: | The proposed models are able to make predictions on multilingual tasks with different tokenizers and model sizes. |