Papers by Jason Baldridge
Where Do We Go From Here? Multi-scale Allocentric Relational Inferencefrom Natural Spatial Descriptions (2024.eacl-long)
Copied to clipboard
| Challenge: | Current NLP navigation studies focus on egocentric local descriptions that require reasoning over the agent’s local perception. |
| Approach: | They propose to use a dataset to analyse English geospatial instructions to find locations and paths from natural language descriptions. |
| Outcome: | The proposed task and dataset includes 10,404 examples of English geospatial instructions for reaching a target location using map-knowledge. |
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing image captioning datasets have limited cross-modal associations, preventing researchers from examining how inter-modal learning impacts intra-modal tasks. |
| Approach: | They propose to use image captioning data to support multi-modal retrieval training and evaluation to assess the impact of inter-modality learning. |
| Outcome: | The proposed model is able to measure the influence of intra- and inter-modality learning. |
PAWS: Paraphrase Adversaries from Word Scrambling (N19-1)
Copied to clipboard
| Challenge: | Existing paraphrase identification datasets lack sentence pairs with high word overlap without being paraphrases. |
| Approach: | They propose a workflow for generating pairs of sentences with high word overlap . they use controlled word swapping and back translation followed by fluency and paraphrase judgments . |
| Outcome: | The proposed dataset has 108,463 well-formed paraphrase and non-paraphrase pairs with high lexical overlap. |
Into the Unknown: Generating Geospatial Descriptions for New Environments (2024.findings-acl)
Copied to clipboard
| Challenge: | Similar to vision-and-language navigation tasks, the Rendezvous (RVS) task requires reasoning over allocentric spatial relationships using non-sequential navigation instructions and maps. |
| Approach: | They propose a large-scale augmentation method for generating high-quality synthetic data for new environments using readily available geospatial data. |
| Outcome: | The proposed method improves accuracy on unseen and seen environments by 45.83% on the Rendezvous (RVS) task. |
Text Classification with Few Examples using Controlled Generalization (N19-1)
Copied to clipboard
| Challenge: | Current training data for text classification is limited, resulting in limited generalization capacity. |
| Approach: | They propose a feed-forward network that can generalize from unlabeled parsed corpora to produce task-specific semantic vectors. |
| Outcome: | The proposed approach is especially effective in low-data scenarios compared to state-of-the-art methods. |
Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation (P19-1)
Copied to clipboard
| Challenge: | Existing metrics for vision-and-language navigation focus on goal completion rather than the sequence of actions corresponding to the instructions. |
| Approach: | They propose to use a room-to-room dataset to measure the length of instruction followed by agents. |
| Outcome: | The proposed metric outperforms existing metrics for Room-to-Room tasks because it is direct-to goal shortest. |
Learning To Split and Rephrase From Wikipedia Edit History (D18-1)
Copied to clipboard
| Challenge: | Performing split and rephrase tasks is one of the main operations in text simplification, alongside paraphrasing and dropping less salient content. |
| Approach: | They propose to use Wikipedia's edit history to extract a rich new dataset for the task. |
| Outcome: | The proposed model scores 32 BLEU points above the previous best on the WebSplit benchmark. |
MURAL: Multimodal, Multitask Representations Across Languages (2021.findings-emnlp)
Copied to clipboard
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, Jason Baldridge
| Challenge: | Image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages. |
| Approach: | They propose a dual encoder that integrates image-text matching and translation pairs to solve two tasks by learning from billions of pairs. |
| Outcome: | The proposed encoder outperforms ALIGN's cross-modal retrieval performance on well-resourced languages and significantly improves on under-resource languages. |
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Room-Across-Room (RxR) is a vision-and-language navigation dataset that addresses gaps in existing ones by addressing known biases in paths and eliciting more references to visible entities. |
| Approach: | They introduce a new Vision-and-Language Navigation (VLN) dataset that addresses biases in paths and elicits more references to visible entities. |
| Outcome: | The proposed model learns from synchronized pose traces by focusing only on portions of the panorama attended to in human demonstrations. |
Underspecification in Scene Description-to-Depiction Tasks (2022.aacl-main)
Copied to clipboard
| Challenge: | Recent text-to-image generation systems have demonstrated impressive capabilities . recent work focuses on generating images depicting scenes from scene descriptions . |
| Approach: | They propose a conceptual framework to address implicitness, ambiguity and underspecification issues in multimodal image+text systems. |
| Outcome: | The proposed framework addresses key challenges concerning textual and visual ambiguity and risks that may be amplified by ambiguous and underspecified elements. |
Mapping Natural Language Instructions to Mobile UI Action Sequences (2020.acl-main)
Copied to clipboard
| Challenge: | a new problem of grounding natural language instructions to mobile UI actions is emerging . we use a Transformer to extract action phrase tuples from long-range natural language instruction . |
| Approach: | They propose a dataset that pairs English instructions with actions performed by people on a mobile UI emulator. |
| Outcome: | The proposed model achieves 70.59% accuracy on predicting complete ground-truth action sequences in PixelHelp. |
On the Evaluation of Vision-and-Language Navigation Instructions (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing instruction generators have not been evaluated using human wayfinders . BLEU, ROUGE, METEOR and CIDEr are ineffective for evaluating grounded navigation instructions. |
| Approach: | They propose an instruction-trajectory compatibility model that operates without reference instructions to improve wayfinding performance. |
| Outcome: | The proposed model shows the highest correlation with human wayfinding outcomes when scoring individual instructions. |
ImageInWords: Unlocking Hyper-Detailed Image Descriptions (2024.emnlp-main)
Copied to clipboard
Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, Radu Soricut
| Challenge: | generating accurate hyper-detailed image descriptions is challenging for vision-language models trained on web-scraped image-text. |
| Approach: | They propose a data-centric framework for generating hyper-detailed image descriptions using web-scraped image-text. |
| Outcome: | The proposed framework improves on human evaluations on the data, even with only 9k samples. |
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification (D19-1)
Copied to clipboard
| Challenge: | Existing work on adversarial data generation focuses on English . Existing multilingual datasets show effectiveness of deep, multilingual pre-training . |
| Approach: | They propose a dataset of 23,659 human translated PAWS evaluation pairs in six languages . they show the effectiveness of deep, multilingual pre-training while leaving considerable headroom . |
| Outcome: | The proposed model shows that multilingual training and evaluation regimes are more accurate than previous models. |
A Fast, Compact, Accurate Model for Language Identification of Codemixed Text (D18-1)
Copied to clipboard
| Challenge: | a feed-forward network can label codemixed and monolingual text in 100 languages and 100 language pairs. |
| Approach: | They propose a feed-forward network that can provide a language code for every token in a sentence . they show that the model can label both codemixed and monolingual text in 100 languages . |
| Outcome: | The proposed model outperforms previous multilingual approaches in terms of accuracy and speed. |