Papers by Jiayuan Mao
Generating Fine Details of Entity Interactions (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing text-to-image models excel at generating high-quality object-centric images from instructions, but lack of data for complex interactions. |
| Approach: | They propose a multimodal Large Language Models-generated dataset to benchmark and enhance interaction-rich images. |
| Outcome: | The proposed approach improves image quality and automatic and human evaluations show improvements. |
Visually Grounded Neural Syntax Acquisition (P19-1)
Copied to clipboard
| Challenge: | a visually grounded neural syntax learner is an approach for learning syntactic representations without any supervision. |
| Approach: | They propose a visually grounded neural syntax learner that acquires syntax by looking at images and reading captions. |
| Outcome: | The proposed model outperforms unsupervised approaches on the MSCOCO data set . it is more stable with choice of initialization and amount of training data, the authors show . |
Language-Mediated, Object-Centric Representation Learning (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent work has studied the problem of unsupervised object representation learning, though without language. |
| Approach: | They propose language-mediated, Objectcentric Representation Learning (LORL) a paradigm for learning disentangled, objectcentric scene representations from vision and language. |
| Outcome: | The proposed paradigm improves performance of unsupervised object discovery algorithms on two datasets using language. |
Learning Visually-Grounded Semantics from Contrastive Adversarial Samples (C18-1)
Copied to clipboard
| Challenge: | Existing frameworks for grounding distributional representations of texts on the visual domain are limited . effective and efficient grounding of distributional embeddings remains challenging . |
| Approach: | They propose to ground distributional representations of texts on the visual domain using visual-semantic embeddings. |
| Outcome: | The proposed model improves on a diverse set of downstream tasks and defends known-type adversarial attacks. |
Foundation Models Meet Embodied Agents (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | This tutorial will present a systematic overview of recent advances in foundation models for embodied agents . |
| Approach: | This tutorial will present a systematic overview of recent advances in foundation models for embodied agents. |
| Outcome: | This tutorial covers three types of foundation models for embodied agents . |
Learning Language through Grounding (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | This tutorial provides a historical overview of grounding and discusses its use in computational linguistics and in computational language processing. |
| Approach: | They introduce the concept of grounding and discuss future directions and open challenges . they will delve into recent progress in learning lexical semantics, syntax, and complex meanings through various forms of ground. |
| Outcome: | This course will provide an overview of the field of grounding and discuss future directions and challenges related to large language models and scaling. |