Papers by Louis-Philippe Morency
Integrating Multimodal Information in Large Pretrained Transformers (2020.acl-main)
Copied to clipboard
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, Ehsan Hoque
| Challenge: | Recent Transformer-based contextual word representations have shown state-of-the-art performance in multiple disciplines within NLP. |
| Approach: | They propose an attachment to BERT and XLNet that allows them to accept multimodal nonverbal data during fine-tuning. |
| Outcome: | The proposed attachment allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. |
Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions (2020.acl-main)
Copied to clipboard
| Challenge: | Experimental results show that N3 can out-perform previous natural-language based zero-shot learning methods across 4 different zero- shot image classification benchmarks. |
| Approach: | They propose a new paradigm for synthesizing task-specific neural networks from language descriptions and a generic pre-trained model from natural language. |
| Outcome: | The proposed model outperforms natural-language based zero-shot learning methods across 4 zero- shot image classification benchmarks. |
Beyond Additive Fusion: Learning Non-Additive Multimodal Interactions (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal fusion addresses the problem of analyzing spoken words in the multimodal context, including visual expressions and prosodic cues. |
| Approach: | They propose to use multimodal fusion to separate unimodal, bimodal, and trimodal interactions in a multimodal model. |
| Outcome: | The proposed model separates unimodal, bimodal, and trimodal interactions while not degrading predictive performance. |
Difference-Masking: Choosing What to Mask in Continued Pretraining (2023.findings-emnlp)
Copied to clipboard
Alex Wilf, Syeda Akter, Leena Mathur, Paul Liang, Sheryl Mathew, Mengrou Shou, Eric Nyberg, Louis-Philippe Morency
| Challenge: | Existing approaches to masked prediction have shown that deciding what to mask can substantially improve learning outcomes. |
| Approach: | They propose a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain. |
| Outcome: | The proposed masking strategy outperforms baselines on language-only and multimodal video tasks. |
Tutorial on Multimodal Machine Learning (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Multimodal machine learning is a challenging but crucial area with numerous applications in multimedia, affective computing, robotics, finance, HCI, and healthcare. |
| Approach: | This tutorial will describe an updated taxonomy on multimodal machine learning synthesizing its core technical challenges and major directions for future research. |
| Outcome: | The proposed taxonomy synthesizes the core technical challenges and major directions for future research. |
No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures (2020.findings-emnlp)
Copied to clipboard
| Challenge: | a study of spoken language and co-speech gestures shows that it is important to model the long tail of the language-gesture distribution. |
| Approach: | They propose a method that combines adversarial learning with importance sampling to strike a balance between precision and coverage. |
| Outcome: | The proposed method outperforms state-of-the-art methods for gesture generation. |
CMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French (2020.emnlp-main)
Copied to clipboard
AmirAli Bagher Zadeh, Yansheng Cao, Simon Hessner, Paul Pu Liang, Soujanya Poria, Louis-Philippe Morency
| Challenge: | Existing datasets in multimodal language are limited and disproportionately affect native speakers of other languages . authors propose a large-scale dataset for Spanish, Portuguese, German and French . |
| Approach: | They propose a large-scale multimodal language dataset for Spanish, Portuguese, German and French. |
| Outcome: | The proposed dataset is the largest of its kind with 40,000 total labelled sentences . it covers a diverse set topics and speakers and carries supervision of 20 labels including sentiment, emotions, and attributes. |
HOLM: Hallucinating Objects with Language Models for Referring Expression Recognition in Partially-Observed Scenes (2022.acl-long)
Copied to clipboard
| Challenge: | a challenge in building AI systems physically present in the world is partial observability, a problem that exists when the entire state of the environment is not known or available to the system. |
| Approach: | They propose a method to infer object hallucinations for the unobserved part of the environment using large pre-trained language models. |
| Outcome: | The proposed method performs better than state-of-the-art approaches on two datasets for dRER. |
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors (P18-1)
Copied to clipboard
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, Louis-Philippe Morency
| Challenge: | Multimodal research is a growing field of artificial intelligence, and fusion is one of the main research problems. |
| Approach: | They propose a low-rank multimodal fusion method which integrates multiple unimodal representations into one compact multimodal representation. |
| Outcome: | The proposed method achieves competitive results on multimodal sentiment analysis, speaker trait analysis, and emotion recognition tasks while reducing computational complexity. |
Towards Debiasing Sentence Representations (2020.acl-main)
Copied to clipboard
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, Louis-Philippe Morency
| Challenge: | Recent work has shown word-level embeddings reflect and propagate social biases present in training corpora. |
| Approach: | They propose a method to debias word embeddings to reduce biases at sentence level . they hope their work will inspire future research on characterizing and removing biase . |
| Outcome: | The proposed method reduces biases and preserves performance on downstream tasks such as sentiment analysis and natural language understanding. |
Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization (P19-1)
Copied to clipboard
Paul Pu Liang, Zhun Liu, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, Louis-Philippe Morency
| Challenge: | Existing methods to regularize multimodal data are imperfect due to imperfect modalities, missing entries or noise corruption. |
| Approach: | They propose a method to regularize multimodal data by tensor rank minimization . they use correlations between time and modalities to generate low-rank tenses . |
| Outcome: | The proposed model achieves good results across various levels of imperfection. |
Counterfactual Augmentation for Multimodal Learning Under Presentation Bias (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In real-world machine learning systems, labels are often derived from user behaviors that the system wishes to encourage. |
| Approach: | They propose a method for correcting presentation bias using generated counterfactual labels by augmentation of the labels by the user. |
| Outcome: | The proposed method improves performance in an oracle setting compared to uncorrected models and existing bias-correction methods. |
Multimodal Transformer for Unaligned Multimodal Language Sequences (P19-1)
Copied to clipboard
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov
| Challenge: | Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. |
| Approach: | They propose a multimodal model that extends the standard Transformer network to learn representations directly from unaligned multimodal streams. |
| Outcome: | The proposed model outperforms state-of-the-art methods on aligned and non-aligned data. |
SenteCon: Leveraging Lexicons to Learn Human-Interpretable Language Representations (2023.findings-acl)
Copied to clipboard
| Challenge: | In many settings, it is important to understand a model’s decision-making process. |
| Approach: | They propose a method for introducing human interpretability in deep language representations by encoding a passage of text as a layer of interpretable categories. |
| Outcome: | The proposed method outperforms existing interpretable language representations on downstream tasks and on agreement with human characterizations of the text. |
Think Twice: Perspective-Taking Improves Large Language Models’ Theory-of-Mind Capabilities (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances to LLMs’ reasoning capabilities from simple yet effective prompting techniques such as Chain-of-Thought have seen limited applicability to ToM. |
| Approach: | They propose a two-stage prompting framework inspired by Simulation Theory's notion of perspective-taking to elicit Theory-of-Mind capabilities in Large Language Models. |
| Outcome: | The proposed framework shows that it is much more effective than existing prompts. |
MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences (2021.naacl-main)
Copied to clipboard
Jianing Yang, Yongxin Wang, Ruitao Yi, Yuying Zhu, Azaan Rehman, Amir Zadeh, Soujanya Poria, Louis-Philippe Morency
| Challenge: | a novel graph-based neural model for multimodal sequential data is proposed . fusion is the process of blending information from multiple modalities, usually preceded by alignment . |
| Approach: | They propose a graph-based neural model that converts unaligned data into a modal-temporal graph . they use a dynamic pruning and read-out technique to efficiently process the graph fusion operation . |
| Outcome: | The proposed model performs state-of-the-art on multimodal sentiment analysis and emotion recognition benchmarks while utilizing significantly fewer model parameters. |
Text-Transport: Toward Learning Causal Effects of Natural Language (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for causal inference require strong assumptions about the data, meaning the data from which one *can* estimate valid causal effects is not representative of the actual target domain of interest. |
| Approach: | They propose a method for estimation of causal effects from natural language under any text distribution using the notion of distribution shift. |
| Outcome: | The proposed method can be used to estimate causal effects from natural language under any text distribution. |
Social Genome: Grounded Social Reasoning Abilities of Multimodal Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Social reasoning is a core competency of social intelligence and requires specialized neural and cognitive systems to be able to interpret multimodal interactions. |
| Approach: | They propose to use social reasoning traces to generate fine-grained explanations using external knowledge. |
| Outcome: | The proposed model is based on 272 videos of human interactions and 1,486 human-annotated reasoning traces related to inferences about these interactions. |
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis (2022.findings-emnlp)
Copied to clipboard
Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, Louis-Philippe Morency
| Challenge: | Pre-trained language models (PLMs) have gained increasing popularity due to compelling prediction performance in diverse natural language processing tasks. |
| Approach: | They compare three popular options for encoding and Temp Scaling for PLMs . they recommend using Temp Loss as uncertainty quantifier and Focal Loss for fine-tuning . |
| Outcome: | Using pre-trained language models, we compare three options on NLP classification tasks and domain shift. |
Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel (D19-1)
Copied to clipboard
| Challenge: | Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction. |
| Approach: | They propose a new formulation of attention via the lens of the kernel which allows us to understand individual components of Transformer's attention. |
| Outcome: | The proposed model outperforms existing models on language understanding and sequence prediction tasks and is more efficient than existing models. |
Visual Referring Expression Recognition: What Do Systems Actually Learn? (N18-2)
Copied to clipboard
| Challenge: | Existing systems for referring expression recognition ignore linguistic structure, instead relying on shallow correlations introduced by unintended biases in the data selection and annotation process. |
| Approach: | They propose to use a system trained on the input image without the input referring expression to achieve a precision of 71.2% in top-2 predictions. |
| Outcome: | The proposed model can achieve 71.2% accuracy on the input image without the input referring expression and 84.2% on the object category given the input. |
Refer360∘: A Referring Expression Recognition Dataset in 360∘ Images (2020.acl-main)
Copied to clipboard
| Challenge: | Refer360° is a large-scale referring expression recognition dataset consisting of 17,137 instruction sequences and ground-truth actions for completing these instructions in 360° scenes. |
| Approach: | They propose a large-scale referring expression recognition dataset, Refer360°, consisting of 17,137 instruction sequences and ground-truth actions for completing these instructions in 360° scenes. |
| Outcome: | The proposed dataset contains 17,137 instruction sequences and ground-truth actions for referring expression recognition in 360° scenes. |
StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer (2021.naacl-main)
Copied to clipboard
Yiwei Lyu, Paul Pu Liang, Hai Pham, Eduard Hovy, Barnabás Póczos, Ruslan Salakhutdinov, Louis-Philippe Morency
| Challenge: | Existing methods for text style transfer focus on individual high-level semantic changes but do not offer fine-grained control of sentence structure, emphasis, and content. |
| Approach: | They propose a large-scale text style transfer benchmark with 21 fine-grained stylistic changes across atomic lexical, syntactic, semantic, and thematic transfers. |
| Outcome: | The proposed method allows modeling fine-grained changes as building blocks for more complex, high-level transfers. |
MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal models focus on the correspondence between images and text, but this only covers a subset of real-world interactions. |
| Approach: | They propose an approach to enhance multimodal models by training separate expert models for each type of interaction, such as redundancy present in both modalities, uniqueness in one modality, or synergy that emerges when both . modality is used to capture overlaps in semantic content between images and text, making a strong multi-view redundancies assumption. |
| Outcome: | The proposed approach improves on a sarcasm detection and humor detection task. |
Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent multimodal learning models with strong performances on human-centric tasks are often black-box with very limited interpretability. |
| Approach: | They propose a multimodal routing algorithm which dynamically adjusts weights between input and output modalities for each input sample. |
| Outcome: | The proposed model can interpret modality-prediction relationships globally and locally for each input sample while keeping competitive performance compared to state-of-the-art methods. |
Strong and Simple Baselines for Multimodal Utterance Embeddings (N19-1)
Copied to clipboard
| Challenge: | Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations. |
| Approach: | They propose two simple but strong baselines to learn embeddings of multimodal utterances by factorizing the utterant into unimodal factors. |
| Outcome: | The proposed models show that they can be derived in closed form while maintaining simplicity and efficiency during learning and inference. |
Global Reward to Local Rewards: Multimodal-Guided Decomposition for Improving Dialogue Agents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for asynchronous dialogue agents only use a single global score at the end of the session. |
| Approach: | They propose a method for aligning an LLM-based dialogue agent for long-term social dialogue . they use local implicit feedback to decompose a human-provided global Explicit reward . |
| Outcome: | The proposed approach improves the turn-level utterance generation across conversational metrics compared to baseline methods. |
Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment (2023.acl-long)
Copied to clipboard
| Challenge: | Despite recent progress towards scaling up multimodal vision-language models, these models struggle on compositional generalization benchmarks such as Winoground. |
| Approach: | They propose to use a cross-modal attention regularization loss to enforce relation alignment by capturing the semantic relation ‘in’ to match the visual attention from the mug to the grass. |
| Outcome: | The proposed approach improves Winoground Group score by 5.75 points . |
Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions (2023.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to de-bias pre-trained large language models focus on changes to training regime, but this is not feasible. |
| Approach: | They propose to de-bias a pre-trained model by fine-tuning it on only 10 examples . they show that the technique performs better than competitive baselines . |
| Outcome: | The proposed method performs better than competitive state-of-the-art baselines with minimal loss in language modeling ability. |
Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data (2021.acl-long)
Copied to clipboard
Paul Pu Liang, Terrance Liu, Anna Cai, Michal Muszynski, Ryo Ishii, Nick Allen, Randy Auerbach, David Brent, Ruslan Salakhutdinov, Louis-Philippe Morency
| Challenge: | Mental health conditions remain underdiagnosed in many countries despite access to advanced medical care . a new approach to learn mood markers from mobile data is needed to improve accuracy and improve learning from typed text. |
| Approach: | They propose to use mobile data to learn mood markers without identifying users through personal or protected attributes. |
| Outcome: | The proposed model obfuscates user identities while remaining predictive . future directions include better models and pre-learning from typed text . |
Multimodal Language Analysis with Recurrent Multistage Fusion (D18-1)
Copied to clipboard
| Challenge: | Comprehending multimodal language requires modeling interactions between modalities and between them. |
| Approach: | They propose a multistage fusion network which decomposes the fusion problem into multiple stages, each focused on a subset of multimodal signals for specialized, effective fusion. |
| Outcome: | The proposed model performs state-of-the-art across three datasets relating to multimodal sentiment analysis, emotion recognition, and speaker traits recognition. |
Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decomposition (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a large language model is used to decompose global feedback into a lightweight reward model. |
| Approach: | They propose a large language model based reward decomposition framework for dialogue agents . they use a frozen large language modeling framework to decompose global feedback . |
| Outcome: | The proposed framework infers fine-grained local rewards from a single session-level feedback signal. |
Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for controlling the distribution of generated text only work with quantified distributions, which require pre-defined categories, proportions of the distribution, or an existing corpus following the desired distributions. |
| Approach: | They propose a few-shot human-in-the-loop training algorithm that allows distribution control for text generation via human feedback. |
| Outcome: | The proposed algorithm achieves state-of-the-art results on single topic/attribute and quantified distribution control compared to previous works. |
Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Building socially-intelligent AI agents involves creating agents that can sense, perceive, reason about, learn from, and respond to affect, behavior, and cognition of other agents. |
| Approach: | They propose a set of technical challenges and open questions for researchers to advance Social-AI. |
| Outcome: | The proposed frameworks are based on the social intelligence competencies that evolved over thousands of years in Homo sapiens and are expected to be the foundations for the development of social-intelligent AI agents. |
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph (P18-1)
Copied to clipboard
| Challenge: | Analyzing human multimodal language is emerging area of research in NLP. |
| Approach: | They propose a multimodal fusion technique to exploit how modalities interact in multimodal language. |
| Outcome: | The proposed technique exploits how modalities interact with each other in human multimodal language. |
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor (D19-1)
Copied to clipboard
Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, Mohammed (Ehsan) Hoque
| Challenge: | Humor is a unique and creative communicative behavior often displayed during social interactions. |
| Approach: | They present a dataset that allows to model multimodal language used in expressing humor using text, visual and acoustic communication. |
| Outcome: | The proposed framework opens the door to understanding multimodal language used in expressing humor. |
Conversational Memory Network for Emotion Recognition in Dyadic Dialogue Videos (N18-1)
Copied to clipboard
Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, Roger Zimmermann
| Challenge: | Existing methods for recognizing emotions in conversations ignore inter-speaker dependency relations . dyadic conversations are a form of dialogue between two entities . |
| Approach: | They propose a deep neural framework which leverages contextual information from the conversation history to model past utterances of each speaker into memories. |
| Outcome: | The proposed framework improves by 3 4% over the state-of-the-art in recognizing emotions in dyadic conversational videos. |