Papers by Noah Constant
FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation (2023.tacl-1)
Copied to clipboard
Parker Riley, Timothy Dozat, Jan A. Botha, Xavier Garcia, Dan Garrette, Jason Riesa, Orhan Firat, Noah Constant
| Challenge: | a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation is presented . FRMT is a type of style-targeted translation that uses labeled training data to perform tasks. |
| Approach: | They propose a dataset and evaluation benchmark for Few-shot Region-aware Machine Translation. |
| Outcome: | The proposed model is based on two translations from English into Portuguese and Mandarin Chinese. |
Multilingual Universal Sentence Encoder for Semantic Retrieval (2020.acl-demos)
Copied to clipboard
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, Ray Kurzweil
| Challenge: | Using a multi-task trained dual-encoder, our models embed text from 16 languages into a shared semantic space. |
| Approach: | They propose retrieval focused multilingual sentence embedding models on TensorFlow Hub. |
| Outcome: | The models achieve state-of-the-art on monolingual and cross-lingual retrieval (SR) and retrieval question answering (ReQA) competitive performance is obtained on related tasks of translation pair bitext retrieval and retrieving question answering. |
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can be more efficient when they are larger than they are in their size. |
| Approach: | They propose a prompt-based transfer learning approach called SPoT: Soft Prompt Transfer that learns a soft prompt on one or more source tasks and initializes it for a target task. |
| Outcome: | The proposed approach outperforms Prompt Tuning and MODELTUNING on superGLUE benchmarks while using up to 27,000 fewer task-specific parameters. |
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2024.findings-acl)
Copied to clipboard
Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, Thang Luong
| Challenge: | Modern large language models often "hallucinate" plausible but factually incorrect information, which reduces their trustworthiness especially in settings where accurate and up-to-date information is critical. |
| Approach: | They develop a human evaluation procedure to measure correctness and hallucination and use it to benchmark both closed and open-source LLMs. |
| Outcome: | The proposed method outperforms both competing search engine-augmented prompting methods and commercial systems on search-augmented QA. |
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Sentence embeddings are useful for language processing tasks, but it is unclear how to produce them from encoder-decoder models. |
| Approach: | They investigate the effects of scaling up sentence encoders to 11B parameters on sentence embeddings from text-to-text transformers (T5) . |
| Outcome: | The proposed models outperform the previous best models on both SentEval and SentGLUE transfer tasks. |
Universal Sentence Encoder for English (D18-2)
Copied to clipboard
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, Ray Kurzweil
| Challenge: | TensorFlow Hub sentence embedding models have good task transfer performance . model variants allow for trade-offs between accuracy and compute resources . |
| Approach: | They propose easy-to-use TensorFlow Hub sentence embedding models with good task transfer performance. |
| Outcome: | The proposed models outperform models without transfer learning and those that use only word-level transfer on a number of NLP tasks. |
ReQA: An Evaluation for End-to-End Answer Retrieval Models (D19-58)
Copied to clipboard
| Challenge: | Popular QA benchmarks like SQuAD have driven progress on identifying answer spans within a specific passage . retrieving relevant answers from a huge corpus of documents is still a challenging problem . |
| Approach: | They propose a benchmark for evaluating large-scale sentence-level answer retrieval models . they establish baselines using both neural encoding models and classical retrieval techniques . |
| Outcome: | The proposed model outperforms human models on identifying answer spans within a specific passage . the proposed model is scalable and can bypass the typical document retrieval step . |
TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text style transfer require style-labeled training data, but use only labeled data at inference time. |
| Approach: | They propose a method that uses readily-available unlabeled text to train style transfer . they use a style vector to condition a decoder to perform style transfer using unlabelled text . |
| Outcome: | The proposed method is competitive on sentiment transfer, even compared to models trained fully on labeled data. |
nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? (2021.acl-short)
Copied to clipboard
| Challenge: | Recent studies have shown that cross-lingual transfer learning in pre-trained multilingual models could be improved further by incorporating parallel data. |
| Approach: | They propose to integrate parallel data into mT5 pre-training to improve results on downstream multilingual and cross-lingual tasks. |
| Outcome: | The proposed model improves cross-lingual transfer significantly in small fine-tuning datasets and small model sizes. |
LAReQA: Language-Agnostic Answer Retrieval from a Multilingual Pool (2020.emnlp-main)
Copied to clipboard
| Challenge: | LAReQA tests for “strong” cross-lingual alignment, requiring semantically related cross-language pairs to be closer in representation space than unrelated same-language pair. |
| Approach: | They propose a new benchmark for language-agnostic answer retrieval from a multilingual candidate pool that tests for "strong" cross-lingual alignment . they augment training data via machine translation and find that model performance is improved by augmenting training data through machine translation . |
| Outcome: | The proposed task is based on multilingual BERT (mBERT) and XLM-R. |
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021.emnlp-main)
Copied to clipboard
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, Melvin Johnson
| Challenge: | Recent advances in multilingual natural language processing have improved performance on benchmarks such as XTREME and XGLUE by 13 points . however, improvements have been easier to achieve in some tasks than others . |
| Approach: | They extend XTREME to XTRAME-R, which includes ten natural language understanding tasks and covers 50 typologically diverse languages. |
| Outcome: | The proposed framework improves the performance on the XTREME multilingual benchmark by 13 points compared to human-level performance on English transfer learning. |
Character-Aware Models Improve Visual Text Rendering (2023.acl-long)
Copied to clipboard
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, Rj Mical, Mohammad Norouzi, Noah Constant
| Challenge: | Current image generation models struggle to produce well-formed visual text due to lack of character-level input features. |
| Approach: | They conduct a series of experiments to compare character-aware vs. character-blind text encoders to determine their spelling ability. |
| Outcome: | The character-aware models outperform character-blind models on a range of novel text rendering tasks. |
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to multilingual machine translation rely on training models on monolingual data for all languages in a multitask setup. |
| Approach: | They propose a vocabulary adaptation scheme to extend the language capacity of multilingual machine translation models by combining monolingual data with a dictionary. |
| Outcome: | The proposed model improves on existing models by preserving the original model and allowing for competitive performance even with only monolingual data. |
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | generative multilingual models fine-tuned on English forget to generate non-English data when labeled data is only available in English . generative models fine tuned on English fail to generate multilingual summarization tasks when labeling data is available in other languages . |
| Approach: | They propose to use prompt tuning to overcome catastrophic forgetting in a generative task in . they assume a strict setting with no parallel data or machine translation . |
| Outcome: | The proposed method can overcome catastrophic forgetting to enable zero-shot cross-lingual generation. |
The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. |
| Approach: | They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. |
| Outcome: | The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters. |
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2022.tacl-1)
Copied to clipboard
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel
| Challenge: | a number of pre-trained language models use sequences of tokens corresponding to word units . token-free models that operate directly on raw text have many advantages . |
| Approach: | They propose a standard Transformer architecture that can be used to process byte sequences . they also characterize trade-offs in terms of parameter count, training FLOPs, and inference speed . |
| Outcome: | The proposed model is more robust to noise and more robust on spelling and pronunciation tasks. |
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2021.naacl-main)
Copied to clipboard
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel
| Challenge: | Current natural language processing pipelines often use transfer learning, where a model is pre-trained on a data-rich task before being fine-tuned on . this significantly limits their use given that roughly 80% of the world population does not speak English. |
| Approach: | They introduce a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. |
| Outcome: | The proposed model achieves state-of-the-art on multilingual benchmarks and a simple technique to prevent accidental translation in the zero-shot setting. |