Papers by Miguel Eckstein
Diagnosing Vision-and-Language Navigation: What Really Matters (2022.naacl-main)
Copied to clipboard
Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Wang, Qi Wu, Miguel Eckstein, William Yang Wang
| Challenge: | Existing models claim to be able to align object tokens with specific visual targets, but there are non-negligible gaps between the two. |
| Approach: | They conduct diagnostic experiments to examine how the agents perceive multimodal input by ablation diagnostics input data. |
| Outcome: | The results show that indoor and outdoor navigation agents refer to object and direction tokens when making decisions. |
Imagination-Augmented Natural Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for Natural Language Understanding focus on textual signals, which hinders models from learning efficiently from limited data samples. |
| Approach: | They propose an Imagination-Augmented Cross-modal Encoder to solve natural language understanding tasks from a novel learning perspective. |
| Outcome: | The proposed learning paradigm bridges the gap between human and agent language understanding in both linguistic and perceptual procedures. |
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing evaluation methods for natural language generation rely on token-level or embedding-level comparisons with text references. |
| Approach: | They propose to use text-to-image generator to generate an image as the embodied imagination for the text snippet and compute the imagination similarity using contextual embeddings. |
| Outcome: | The proposed metric improves existing evaluation metrics’ correlations with human similarity judgments in both reference-based and reference-free scenarios. |
Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that GPT-k models focus more on inserting modifiers than predicting spontaneous changes in the primary subject matter. |
| Approach: | They compare the common edits made by humans and GPT-k models to examine their performance in prompting T2I. |
| Outcome: | The proposed models improve the prompt editing process by 20-30%, the authors show . they show that humans tend to replace words and phrases with modifiers . |
SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Iterative language-based image editing (ILBIE) tasks follow iterative instructions to edit images step by step. data scarcity makes learning the association between vision and language challenging. |
| Approach: | They propose a framework that incorporates counterfactual thinking to overcome data scarcity by combining out-of-distribution instructions with previous images. |
| Outcome: | The proposed model improves the correctness of ILBIE on two IBLIE datasets, even with only 50% of the training data. |
Visualize Before You Write: Imagination-Guided Open-Ended Text Generation (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing tools for text-to-image synthesis can visualize machine imaginations for a given context. |
| Approach: | They propose a framework that uses machine-generated images to guide language models in open-ended text generation. |
| Outcome: | The proposed framework is effective on open-ended text generation tasks while showing minor degeneration. |