Papers by Hung Le
LAVIS: A One-stop Library for Language-Vision Intelligence (2023.acl-demo)
Copied to clipboard
| Challenge: | a new open-source library for language-vision research and applications is available for free. |
| Approach: | They introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications. |
| Outcome: | The proposed library is open-source and highly extensible and configurable. |
Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems (P19-1)
Copied to clipboard
| Challenge: | Existing work on video-grounded dialogue systems is limited by feature space and semantic information. |
| Approach: | They propose multimodal transformer networks to encode videos and incorporate information from different modalities. |
| Outcome: | The proposed system generates appropriate conversational response to queries of humans based on visual and audio aspects of a given video . it also generalizes to another multimodal visual-grounded dialogue task, and obtains promising performance. |
Do Domain-specific Experts exist in MoE-based LLMs? (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on domain-specific experts in Large Language Models (LLMs) are still lacking. |
| Approach: | They propose a training-free framework that introduces zero additional inference cost and outperforms well-trained MoE-based LLMs. |
| Outcome: | The proposed framework outperforms well-trained MoE-based LLMs and strong baselines across target and non-target domains. |
Multimodal Dialogue State Tracking (2022.naacl-main)
Copied to clipboard
| Challenge: | Dialogue state tracking is a key component of dialogue systems. |
| Approach: | They propose to extend the definition of dialogue state tracking to multimodality . they propose a new synthetic benchmark and a novel baseline for this task . |
| Outcome: | The proposed task is based on a synthetic benchmark and a self-supervised video understanding task. |
CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | coding tasks require generated code to be fully executable and functionally correct . current agentic approaches struggle with multi-stage planning, generating, and debugging . |
| Approach: | They propose a framework for LLM agents to efficiently explore the search space in different stages of the code generation process. |
| Outcome: | The proposed framework achieves top results on 7 code generation benchmarks and a 31.9% solving rate on the SWEBench benchmark. |
VGNMN: Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems (2022.naacl-main)
Copied to clipboard
| Challenge: | Neural module networks (NMN) have been used in image-grounded tasks such as Visual Question Answering (VQA) however, very limited work on NMN has been studied in the video-ground dialogue tasks. |
| Approach: | They propose to use video as the grounding feature in video-grounded dialogues to model the information retrieval process in videogrounded language tasks as a pipeline of neural modules. |
| Outcome: | The proposed model can achieve promising performance on video-grounded dialogue and QA benchmarks. |
UniConv: A Unified Conversational Neural Architecture for Multi-domain Task-oriented Dialogues (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to training dialogue agents separately are not optimized for multi-domain task-oriented dialogues. |
| Approach: | They propose a unified neural architecture for end-to-end conversational systems in multi-domain task-oriented dialogues that jointly trains a bi-level state tracker and a joint dialogue act and response generator. |
| Outcome: | The proposed system outperforms existing systems on the MultiWOZ2.1 benchmark in dialogue state tracking, context-to-text, and end-to end settings. |
Sample Efficient Alignment Learning With Episodic Control (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing parametric methods for aligning large language models with task objectives are limited. |
| Approach: | They propose a non-parametric framework that aligns large language models with task objectives . they use a key-value memory to store associations between generated text and its corresponding values . |
| Outcome: | The proposed framework outperforms state-of-the-art baselines on harmless, helpful, and summarization tasks. |
SimSMoE: Toward Efficient Training Mixture of Experts via Solving Representational Collapse (2025.findings-naacl)
Copied to clipboard
| Challenge: | Sparse mixture of experts (SMoE) is an effective approach for scaling large language models with a constant computational cost. |
| Approach: | They propose a similarity-based sparse mixture of experts (SimSMoE) that guarantees a solution to address the representation collapse problem between experts given a fixed FLOPs budget. |
| Outcome: | The proposed method outperforms existing methods and outperformed other SMoE routing methods in performance. |
DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue (2021.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks do not have enough annotations to analyze video-grounded dialogue systems and understand their capabilities and limitations in isolation. |
| Approach: | They present a Diagnostic Dataset for Video-grounded dialogue with minimal biases and detailed annotations for the different types of reasoning over the spatio-temporal space of video. |
| Outcome: | The proposed system is based on 11k CATER synthetic videos and contains 10 instances of 10-round dialogues for each video. |
SPaCe: Unlocking Sample-Efficient Large Language Models Training With Self-Pace Curriculum Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing training pipelines sample training examples uniformly across steps or epochs, ignoring differences in difficulty, redundancy, and learning value, which slows learning and wastes computation. |
| Approach: | They propose a self-paced learning framework that enables efficient learning based on the capability of the model being trained through optimizing which data to use and when. |
| Outcome: | The proposed framework achieves comparable or better accuracy than state-of-the-art baselines while using up to (100 times) fewer samples. |
Video-Grounded Dialogues with Pretrained Generation Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Pre-trained language models have shown success in improving downstream NLP tasks . pre-tuned models capture textual dependencies in text data of rich semantics . |
| Approach: | They propose a framework for improving video-grounded dialogue by extending GPT-2 models . they propose to combine visual and textual representation into a structured sequence . |
| Outcome: | The proposed framework improves audio-visual scene-aware dialogues benchmark on AVSD . it is based on a large pre-trained GPT-2 network and can generate natural responses . |
Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization (2026.acl-long)
Copied to clipboard
Yifeng Ding, Hung Le, Songyang Han, Kangrui Ruan, Zhenghui Jin, Varun Kumar, Zijian Wang, Anoop Deoras
| Challenge: | Current reinforcement learning methods suffer from coarse-grained, trajectory-level rewards that provide insufficient learning signals for complex multi-turn interactions, leading to training stagnation. |
| Approach: | They propose a novel RL algorithm for training large language models for multi-turn tool-integrated reasoning (TIR) that incorporates three innovations: turn-level reward assignment that provides fine-grained feedback for individual turns, return-based advantage estimation where normalized discounted returns are calculated as advantages, and self-supervised reward shaping that exploits self-supervision signals from generated code to densify sparse binary outcome-based rewards. |
| Outcome: | The proposed algorithm outperforms GRPO by 3.0% across diverse math reasoning benchmarks and improves grepo by 3.9% on commonsense reasoning and program synthesis tasks. |
SuperRAG: Beyond RAG with Layout-Aware Graph Modeling (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing methods that only deal with flat text chunks use a graph structure to handle complex questions. |
| Approach: | They propose layout-aware graph modeling for multimodal RAG using document layout parsing to take into account relationship of multimodalities. |
| Outcome: | The proposed method can handle complex questions that require information from multimodalities. |
BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to video-grounded dialogues focus on superficial temporal-level visual cues, but neglect more fine-grained spatial signals from videos. |
| Approach: | They propose a vision-language neural framework for high-resolution queries in videos based on textual cues that exploits both spatial and temporal-level information. |
| Outcome: | The proposed approach outperforms previous approaches on the TGIF-QA benchmark and significantly outperformed previous approaches. |
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing code LLMs adopt a specific architecture or rely on a unified encoder-decoder network for downstream tasks, lacking flexibility to operate in the optimal architecture for a particular task. |
| Approach: | They propose to initialize code LLMs with frozen off-the-shelf LLM and explore instruction-tuning to align with natural language instructions. |
| Outcome: | The proposed model outperforms open-source LLMs on 20 code-related benchmarks. |
Dynamic Steering With Episodic Memory For Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing activation steering methods apply a single sentence-level steering vector uniformly across all tokens, ignoring LLMs’ token-wise, auto-regressive nature. |
| Approach: | They propose a framework that aligns LLMs to given demonstrations by steering at the token level conditioned on the input query. |
| Outcome: | The proposed framework surpasses baselines across safety, style transfer, and role-playing tasks, demonstrating improved alignment as demonstration scales. |
Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback (2022.findings-naacl)
Copied to clipboard
Duy-Hung Nguyen, Nguyen Viet Dung Nghiem, Bao-Sinh Nguyen, Dung Tien Tien Le, Shahab Sabahi, Minh-Tien Nguyen, Hung Le
| Challenge: | a framework to train summarization models with preference feedback is proposed . human-in-the-loop (HITL) allows humans to actively participate in supervising AI systems . |
| Approach: | They propose a framework to train summarization models with preference feedback interactively. |
| Outcome: | The proposed framework improves ROUGE scores and sample-efficiency in active, few-shot and online settings. |