Papers by Armen Aghajanyan
Conversational Semantic Parsing (2020.emnlp-main)
Copied to clipboard
Armen Aghajanyan, Jean Maillard, Akshat Shrivastava, Keith Diedrick, Michael Haeger, Haoran Li, Yashar Mehdad, Veselin Stoyanov, Anuj Kumar, Mike Lewis, Sonal Gupta
| Challenge: | Structured representations for task-oriented assistant systems are limited due to the limitations of the representation. |
| Approach: | They propose a semantic representation for task-oriented conversational systems that can represent co-reference and context carryover. |
| Outcome: | The proposed model improves the best results on ATIS, SNIPS, TOP and DSTC2 by up to 5 points for slot-carryover. |
Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog (2021.naacl-main)
Copied to clipboard
| Challenge: | Semantic parsing using sequence-to-sequence models is stymied by higher compute requirements and higher latency. |
| Approach: | They propose a non-autoregressive approach to predict semantic parse trees with an efficient seq2seq model architecture. |
| Outcome: | The proposed architecture achieves an 81% reduction in latency on TOP dataset and retains competitive performance over non-pretrained models on three different semantic parsing datasets. |
CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to answer open domain questions rely on unlabeled text or synthetically generated question-answer pairs. |
| Approach: | They propose a large-scale open-domain question-answering dataset based on the Common Crawl project that can be used to in-domain pre-train popular language models. |
| Outcome: | The proposed dataset achieves promising results in zero-shot, low resource and fine-tuned settings across multiple tasks, models and benchmarks. |
Muppet: Massive Multi-task Representations with Pre-Finetuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows gains from pre-training and fine-tuning that are multi-task . but it can be difficult to know which intermediate tasks will best transfer . |
| Approach: | They propose a large-scale learning stage for pre-finetuning between pre-training and fine-tun. |
| Outcome: | The proposed model improves performance on pretrained discriminators and generation models on a wide range of tasks while improving sample efficiency during fine-tuning. |
Towards Language Agnostic Universal Representations (P19-1)
Copied to clipboard
| Challenge: | Current representations in machine learning are language dependent . however, fluent bilingual speakers rarely face trouble translating a task learned in one language to another . |
| Approach: | They propose a method to decouple the language from the problem by learning language agnostic representations. |
| Outcome: | The proposed model achieves similar accuracies in a single language and in another language. |
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models can be fine-tuned to produce state-of-the-art results for a wide range of language understanding tasks. |
| Approach: | They propose to analyze fine-tuning through the lens of intrinsic dimension . they show that pre-trained models have a low intrinsic dimension reparameterization . |
| Outcome: | The proposed model can achieve 90% of the full parameter performance levels on MRPC with low data regime. |
Improving Passage Retrieval with Zero-Shot Question Generation (2022.emnlp-main)
Copied to clipboard
Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, Luke Zettlemoyer
| Challenge: | Existing re-ranking methods for open-domain question answering are not domain- or task-specific. |
| Approach: | They propose a simple and effective re-ranking method for improving passage retrieval in open-domain question answering. |
| Outcome: | The proposed method outperforms strong supervised models on open-domain questions and triviaQA datasets on top-1000 passages. |
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021.emnlp-main)
Copied to clipboard
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, Christoph Feichtenhofer
| Challenge: | Recent work adopts a "pre-training + fine-tuning" approach for zero-shot transfer to end tasks without fine- tuning. |
| Approach: | They propose a contrastive approach to pre-train a transformer model for zero-shot video and text understanding without using any labels on downstream tasks. |
| Outcome: | The proposed model outperforms supervised approaches on downstream tasks and outperformed previous approaches. |