Papers by Lawrence Jang
MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal models focus on the correspondence between images and text, but this only covers a subset of real-world interactions. |
| Approach: | They propose an approach to enhance multimodal models by training separate expert models for each type of interaction, such as redundancy present in both modalities, uniqueness in one modality, or synergy that emerges when both . modality is used to capture overlaps in semantic content between images and text, making a strong multi-view redundancies assumption. |
| Outcome: | The proposed approach improves on a sarcasm detection and humor detection task. |
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024.acl-long)
Copied to clipboard
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, Daniel Fried
| Challenge: | Existing benchmarks focus on text-based agents, neglecting many natural tasks that require visual information to effectively solve. |
| Approach: | They propose a benchmark to assess the performance of multimodal web agents . they use visual and textual inputs to process and interpret natural language instructions . |
| Outcome: | a new benchmark assesses the performance of multimodal agents on visually grounded tasks . the benchmark identifies limitations of text-only agents and offers insights towards building stronger agents for the web . |