Papers by Steven Hoi
Towards Low-Resource Automatic Program Repair with Meta-Learning and Pretrained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep learning (DL) based APR models have demonstrated promising results by learning from large-scale bug-fix examples in a data-driven manner. |
| Approach: | They propose a meta-learning framework integrated with code pretrained language models to generate fixes for low-resource bugs with limited training samples. |
| Outcome: | The proposed framework learns better error-specific knowledge from high-resource bugs through efficient first-order meta-learning optimization, which allows for a faster adaptation to the target low-resourced bugs. |
Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems (P19-1)
Copied to clipboard
| Challenge: | Existing work on video-grounded dialogue systems is limited by feature space and semantic information. |
| Approach: | They propose multimodal transformer networks to encode videos and incorporate information from different modalities. |
| Outcome: | The proposed system generates appropriate conversational response to queries of humans based on visual and audio aspects of a given video . it also generalizes to another multimodal visual-grounded dialogue task, and obtains promising performance. |
Multimodal Dialogue State Tracking (2022.naacl-main)
Copied to clipboard
| Challenge: | Dialogue state tracking is a key component of dialogue systems. |
| Approach: | They propose to extend the definition of dialogue state tracking to multimodality . they propose a new synthetic benchmark and a novel baseline for this task . |
| Outcome: | The proposed task is based on a synthetic benchmark and a self-supervised video understanding task. |
VGNMN: Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems (2022.naacl-main)
Copied to clipboard
| Challenge: | Neural module networks (NMN) have been used in image-grounded tasks such as Visual Question Answering (VQA) however, very limited work on NMN has been studied in the video-ground dialogue tasks. |
| Approach: | They propose to use video as the grounding feature in video-grounded dialogues to model the information retrieval process in videogrounded language tasks as a pipeline of neural modules. |
| Outcome: | The proposed model can achieve promising performance on video-grounded dialogue and QA benchmarks. |
What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases (2024.naacl-long)
Copied to clipboard
| Challenge: | Vision-language models have broad competence that is difficult to evaluate . current evaluation benchmarks focus on only assessing one or a few capabilities . |
| Approach: | They perform a large-scale transfer learning experiment to discover latent VL skills from data. |
| Outcome: | The results suggest that factor analysis can identify reasonable yet surprising VL skill factors . the results contribute to the design of balanced and broad-coverage vision-language evaluation methods. |
Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that close-sourced LLMs lack the ability to integrate into real-world applications due to their high associated costs and ethical concerns. |
| Approach: | They propose to use student model to refine its own solution by querying ChatGPT to generate task instruction and solution pairs and querying data to refine model. |
| Outcome: | The proposed model outperforms standard distillation with only one third of the data. |
MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments (2026.acl-long)
Copied to clipboard
Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chenglin Cai, Hanzhang Zhou, Jianan Zhang, Liangyu Chen, Zhidan Liu, Steven Hoi, Yue Wang
| Challenge: | AndroidWorld is the dominant mobile GUI agent evaluation benchmark, but its success rates are low . despite reproducible emulator environment, it lacks key application categories such as e-commerce and enterprise communication. |
| Approach: | They propose a benchmark for mobile GUI agents that reflects real-world usage through long-horizon, cross-application workflows. |
| Outcome: | The proposed framework achieves over 90% success rates, while AndroidWorld is the dominant benchmark. |
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing code LLMs adopt a specific architecture or rely on a unified encoder-decoder network for downstream tasks, lacking flexibility to operate in the optimal architecture for a particular task. |
| Approach: | They propose to initialize code LLMs with frozen off-the-shelf LLM and explore instruction-tuning to align with natural language instructions. |
| Outcome: | The proposed model outperforms open-source LLMs on 20 code-related benchmarks. |
HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts (2023.emnlp-main)
Copied to clipboard
Truong Do, Le Khiem, Quang Pham, TrungTin Nguyen, Thanh-Nam Doan, Binh Nguyen, Chenghao Liu, Savitha Ramasamy, Xiaoli Li, Steven Hoi
| Challenge: | Recent studies suggest that fixing the routers can achieve competitive performance by alleviating the collapsing problem, where all experts eventually learn similar representations. |
| Approach: | They propose a method that dynamically generates router parameters through a fixed hypernetwork and trainable embeddings to achieve a balance between training the routers and freezing them to learn an improved routing policy. |
| Outcome: | Experiments on a wide range of tasks show that the proposed method performs better than existing methods. |