Papers by Tsu-Jui Fu
EDIS: Entity-Driven Image Search over Multimodal Web Content (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing image retrieval methods require large datasets and a large candidate set. |
| Approach: | They propose a news-domain dataset for cross-modal image search with 1 million web images . they propose combining multimodal image-text pairs with a million candidates . |
| Outcome: | The proposed dataset challenges state-of-the-art methods with dense entities and the large-scale candidate set. |
ULN: Towards Underspecified Vision-and-Language Navigation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-and-language navigation models are brittle to multi-level language underspecification. |
| Approach: | They propose to use multi-level underspecified instructions to guide agents . they propose to learn GSS for navigation agent to ground multi- level instructions . experimental results show existing VLN models are still brittle to multi-language underspecification . |
| Outcome: | Experimental results show that the proposed framework outperforms baselines on ULN by 10% relative success rate across all levels. |
CPL: Counterfactual Prompt Learning for Vision and Language Models (2022.emnlp-main)
Copied to clipboard
Xuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, Xin Wang
| Challenge: | Existing prompt tuning methods tend to learn spurious or entangled representations, leading to poor generalization to unseen concepts. |
| Approach: | They propose a prompt tuning technique that tunes the learnable prompt for pre-trained vision and language models. |
| Outcome: | The proposed method improves few-shot performance on vision and language tasks over existing prompt tuning methods. |
L2C: Describing Visual Differences Needs Semantic Understanding of Individuals (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for captioning images without understanding individual's semantics are not effective . a new task, visual comparison, has drawn increasing attention in the field of language and vision . |
| Approach: | They propose a learning-to-compare model which learns to understand semantic structures of two images and compares them while learning to describe each one. |
| Outcome: | The proposed model outperforms the baseline and human evaluation on the Birds-to-Words dataset. |
Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that GPT-k models focus more on inserting modifiers than predicting spontaneous changes in the primary subject matter. |
| Approach: | They compare the common edits made by humans and GPT-k models to examine their performance in prompting T2I. |
| Outcome: | The proposed models improve the prompt editing process by 20-30%, the authors show . they show that humans tend to replace words and phrases with modifiers . |
SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Iterative language-based image editing (ILBIE) tasks follow iterative instructions to edit images step by step. data scarcity makes learning the association between vision and language challenging. |
| Approach: | They propose a framework that incorporates counterfactual thinking to overcome data scarcity by combining out-of-distribution instructions with previous images. |
| Outcome: | The proposed model improves the correctness of ILBIE on two IBLIE datasets, even with only 50% of the training data. |
Speed Reading: Learning to Read ForBackward via Shuttle (D18-1)
Copied to clipboard
| Challenge: | LSTM-Shuttle uses human speed reading techniques to perform natural language processing tasks. |
| Approach: | They propose a model which uses human speed reading techniques to perform natural language processing tasks for accurate and efficient comprehension. |
| Outcome: | The proposed model predicts on IMDB, Rotten Tomatoes, AG, and Children’s Book Test datasets and goes backwards. |
GraphRel: Modeling Text as Relational Graphs for Joint Entity and Relation Extraction (P19-1)
Copied to clipboard
| Challenge: | GraphRel is an end-to-end relation extraction model that uses graph convolutional networks to learn named entities and relations. |
| Approach: | They propose a graph-based relation extraction model which uses graph convolutional networks to jointly learn named entities and relations. |
| Outcome: | The proposed model outperforms previous models on two public datasets: NYT and WebNLG. |
Text-guided 3D Human Generation from 2D Collections (2023.findings-emnlp)
Copied to clipboard
| Challenge: | 3D human modeling is used for engaging interaction in gaming, film, and animation. however, the customization of characters is crucial for creativity and scalability. |
| Approach: | They propose a 3D human generation using fashion descriptions to enhance 3D geometry transformation and fine-grained consistency. |
| Outcome: | The proposed model can generate a 3D human, guided by a fashion description, with high efficiency. |
Semi-Supervised Policy Initialization for Playing Games with Language Hints (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for Reinforcement Learning only give out the achieved signal, making learning difficult. |
| Approach: | They propose a semi-supervised initialization that allows the agent to learn from various possible hints before training under different tasks. |
| Outcome: | The proposed method improves learning speed and success rate by 11% compared to other methods. |
H-FND: Hierarchical False-Negative Denoising for Distant Supervision Relation Extraction (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing work on distant supervision denoising introduces false-positive (FP) and falsenegative (FN) training instances to the generated datasets. |
| Approach: | They propose a hierarchical false-negative denoising framework for distant supervision relation extraction that denoises false-positive and false- negative training instances. |
| Outcome: | The proposed framework can revise FN instances correctly and maintain high F1 scores even when 50% of instances have been turned into negatives. |
Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation (2021.eacl-main)
Copied to clipboard
Wanrong Zhu, Xin Wang, Tsu-Jui Fu, An Yan, Pradyumna Narayana, Kazoo Sone, Sugato Basu, William Yang Wang
| Challenge: | Outdoor vision-and-language navigation (VLN) tasks require visual grounding to generate correct actions. |
| Approach: | They propose a multimodal text style transfer learning approach to mitigate data scarcity in outdoor vision-and-language navigation tasks. |
| Outcome: | The proposed approach outperforms baseline models on the outdoor vision-and-language navigation task, improving task completion rate by 8.7% relative to the baseline models. |
TC-Bench: Benchmarking Temporal Compositionality in Conditional Video Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing video generation models struggle to interpret compositional changes and synthesize components across different time steps. |
| Approach: | They propose a temporal compositionality benchmark that uses text prompts and ground truth videos to evaluate compositional changes in video. |
| Outcome: | The proposed benchmark can be used for text-to-video and image-to video generation. |