Papers by Binyuan Hui
An Investigation of LLMs’ Inefficacy in Understanding Converse Relations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for Large Language Models (LLMs) follow the data distribution of pre-training data. |
| Approach: | They propose a benchmark ConvRe focusing on converse relations which contains 17 relations and 1240 triples extracted from popular knowledge graph completion datasets. |
| Outcome: | The proposed benchmark focuses on converse relations, which contains 17 relations and 1240 triples extracted from popular knowledge graph completion datasets. |
CodeArena: Evaluating and Aligning CodeLLMs on Human Preference (2025.emnlp-main)
Copied to clipboard
Jian Yang, Jiaxi Yang, Wei Zhang, Jin Ke, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Zhoujun Li, Binyuan Hui, Junyang Lin
| Challenge: | Code large language models (codeLLMs) focus on synthesizing the correct code snippet, ignoring the alignment with human preferences. |
| Approach: | They propose a benchmark code-based on 40 categories and 44 programming languages to emulate real-world coding tasks. |
| Outcome: | The proposed benchmarks show that open-source code LLMs perform better than open-sourced ones. |
IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web (2025.findings-acl)
Copied to clipboard
Hongcheng Guo, Wei Zhang, Junhao Chen, Yaonan Gu, Jian Yang, Junjia Du, Shaosheng Cao, Binyuan Hui, Tianyu Liu, Jianxin Ma, Chang Zhou, Zhoujun Li
| Challenge: | Existing models have been introduced to improve image comprehension, but there is no robust benchmark for imagetoweb conversion. |
| Approach: | They propose a benchmark to assess imagetoweb conversion proficiency of large multimodal models . they propose to measure layout information of web pages by parsing the Document Object Model tree . |
| Outcome: | The proposed benchmark measures the layout information of web pages—i.e., the positional relationships between elements—which has been overlooked by prior work. |
Turning the Tide: Repository-based Code Reflection (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Code large language models (LLMs) enhance programming by understanding and generating code across languages. |
| Approach: | a new benchmark evaluates code understanding and generation in repositories using code large language models. |
| Outcome: | The proposed model improves code understanding and generation in repositories by evaluating 1,888 test cases across 6 programming languages. |
STAR: SQL Guided Pre-Training for Context-dependent Text-to-SQL Parsing (2022.findings-emnlp)
Copied to clipboard
Zefeng Cai, Xiangyu Li, Binyuan Hui, Min Yang, Bowen Li, Binhua Li, Zheng Cao, Weijie Li, Fei Huang, Luo Si, Yongbin Li
| Challenge: | Extensive experiments show that STAR outperforms previous pre-training methods and ranks first on the leaderboard . text-to-SQL parsing aims to translate natural language (NL) questions into executable SQL queries . |
| Approach: | They propose a SQL guided pre-training framework STAR for context-dependent text-to-SQL parsing . they propose two objectives that explore context-dependence of NL utterances and SQL queries . |
| Outcome: | The proposed framework outperforms existing methods on two downstream benchmarks and ranks first on the leaderboard. |
Out-of-Domain Intent Detection Considering Multi-Turn Dialogue Contexts (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for OOD intent detection are limited to single dialogue turns. |
| Approach: | They propose a context-aware OOD intent detection framework to model multi-turn contexts in OOD context detection tasks using unlabeled data. |
| Outcome: | The proposed framework improves the F1-OOD score by 29% on multi-turn OOD detection tasks compared to the previous best method. |
START: Self-taught Reasoner with Tools (2025.emnlp-main)
Copied to clipboard
Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Bowen Yu, Binyuan Hui, Junyang Lin, Xiang Wang, Dayiheng Liu
| Challenge: | Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in complex reasoning through long chain-of-thought, yet they struggle with precise computations and algorithmic operations. |
| Approach: | They propose a training-free approach that activates LRMs’ latent tool-use capabilities through artificial hints and a framework that enables models to learn effective tool utilization through diverse hint patterns and rejection-based data synthesis. |
| Outcome: | Experiments show that START significantly improves state-of-the-art LRMs across challenging benchmarks, including competition-level mathematics (AMC23: 95.0%, AIME24: 75.6%) and graduate-level science questions (GPQA: 64.6%). |
SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers (2022.coling-1)
Copied to clipboard
Bowen Qin, Lihan Wang, Binyuan Hui, Bowen Li, Xiangpeng Wei, Binhua Li, Fei Huang, Luo Si, Min Yang, Yongbin Li
| Challenge: | Existing methods that learn from multiple semantically-equivalent questions are limited to one-to-one mapping . |
| Approach: | They propose a constraint to explore the underlying complementary semantic information among multiple semantically-equivalent questions and learn robust feature representations with reduced spurious associations. |
| Outcome: | The proposed method outperforms strong competitors and achieves state-of-the-art results on five benchmark datasets. |
PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts (2023.acl-long)
Copied to clipboard
| Challenge: | Existing research on multi-modal dialogue pre-training is limited due to limited availability of multi-dimensional data . a recent emergence of chatGPT 1 has increased confidence in the potential for this goal . |
| Approach: | They propose a framework for multi-modal dialogue pre-training that integrates experts to accommodate multi-faceted tasks. |
| Outcome: | The proposed framework achieves state-of-the-art on eight multi-modal dialog benchmarks. |
One-Shot Learning as Instruction Data Prospector for Large Language Models (2024.acl-long)
Copied to clipboard
Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Ling-Hao Chen, Junhao Liu, Tongliang Liu, Fei Huang, Yongbin Li
| Challenge: | Contemporary practices in instruction tuning often hinge on enlarging data scaling without a clear strategy for ensuring data quality. |
| Approach: | They propose a method that leverages one-shot learning to discern and select high-quality instruction data from extensive datasets. |
| Outcome: | Nuggets outperforms existing methods on MT-Bench and Alpaca-Eval benchmarks. |
TrojanSQL: SQL Injection against Natural Language Interface to Database (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on text-to-SQL systems have not investigated its security aspects . however, how to implement such attacks remains an open question. |
| Approach: | They propose a backdoor-based SQL injection framework for text-to-SQL systems that uses boolean-based injection and union-based injecting techniques to exploit SQL injection vulnerabilities. |
| Outcome: | The proposed framework can produce harmful SQL statements invalidating user queries or compromise sensitive information about the database. |
Synthesizing Text-to-SQL Data from Weak and Strong LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | a capability gap exists between open-source and closed-source large language models (LLMs) . the adoption of closed-sourced LLMs introduces concerns pertaining to openness, privacy, and substantial costs. |
| Approach: | They propose a synthetic data approach that combines strong and weak models for error information . they demonstrate the effectiveness of SENSE, a specialized text-to-SQL model . |
| Outcome: | The proposed method enhances the domain generalization of text-to-SQL models and explores the potential of error data supervision through preference learning. |
Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study (2025.coling-main)
Copied to clipboard
Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, Kai Chen
| Challenge: | Existing benchmarks focused on simplified or isolated aspects of coding, ignoring the full spectrum of programming challenges. |
| Approach: | They propose a case study that examines the performance of large language models across the entire software development lifecycle with four programming languages, multiple domains, and carefully designed and verified metrics for each task. |
| Outcome: | The proposed model performs across the entire software development lifecycle, including design, environment setup, implementation, acceptance testing, and unit testing. |
History Semantic Graph Enhanced Conversational KBQA with Temporal Information Modeling (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for conversational KBQA assume the independence of utterances and model them in isolation. |
| Approach: | They propose a History Semantic Graph Enhanced KBQA model that models long-range semantic dependencies in conversation history while maintaining low computational cost. |
| Outcome: | The proposed model outperforms baselines on a widely used question type dataset. |
Qwen2.5-xCoder: Multi-Agent Collaboration for Multilingual Code Instruction Tuning (2025.acl-long)
Copied to clipboard
Jian Yang, Wei Zhang, Yibo Miao, Shanghaoran Quan, Zhenhe Wu, Qiyao Peng, Liqun Yang, Tianyu Liu, Zeyu Cui, Binyuan Hui, Junyang Lin
| Challenge: | Existing methods to train code LLMs view each programming language in isolation . experimental results show that Qwen2.5-xCoder can bridge the gap between different programming languages . |
| Approach: | They propose a framework that allows agents to collaborate to enhance multilingual instruction tuning for code LLMs. |
| Outcome: | Experimental results show that Qwen2.5-xCoder can transfer knowledge efficiently and effectively between languages. |
SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for dialog understanding only consider self-augmented dialogs as positive samples and treat all other dialogs like negative ones. |
| Approach: | They propose a tree-structured pre-trained conversation model which learns dialog representations from limited labeled dialogs and large-scale unlabeled dialog corpora via semi-supervised contrastive pre-training. |
| Outcome: | The proposed model can achieve state-of-the-art results on the DialoGLUE benchmark. |
S2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing graph-based encoders for text-to-SQL do not model the syntax of natural language questions. |
| Approach: | They propose to inject Syntax to question-Schema graph encoder for text-to-SQL parsers and employ the decoupling constraint to induce diverse relational edge embedding. |
| Outcome: | The proposed approach outperforms all existing methods when pre-training models are used, resulting in a performance ranks first on the Spider leaderboard. |
From Completion to Editing: Unlocking Context-Aware Code Infilling via Search-and-Replace Instruction Tuning (2026.acl-long)
Copied to clipboard
Jiajun Zhang, Zeyu Cui, Jiaxi Yang, Lei Zhang, Yuheng Jing, Zeyao Ma, Tianyi Bai, Zilei Wang, Qiang Liu, Liang Wang, Binyuan Hui, Junyang Lin
| Challenge: | Fill-in-the-Middle (FIM) models suffer from performance degradation and prohibitive latency. |
| Approach: | They propose a search-and-replace infilling framework that integrates agentic verification and editing into a single-pass inference process. |
| Outcome: | The proposed framework harmonizes completion tasks with the instruction-following priors of Chat LLMs, extending the paradigm from static infilling to dynamic context-aware editing. |
Multimodal Recommendation Dialog with Subjective Preference: A New Challenge and Benchmark (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing multimodal task-oriented dialog data fails to demonstrate the diverse expressions of user subjective preferences and recommendation acts in the real-life shopping scenario. |
| Approach: | They propose a multimodal task-oriented dialog dataset with subjective preferences and recommendation acts that is well-annotated with sales experts. |
| Outcome: | The proposed model is powered by a state-of-the-art multimodal model for these tasks. |
Tree-Instruct: A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment (2024.lrec-main)
Copied to clipboard
| Challenge: | Extensive research has highlighted the importance of data complexity as a crucial metric, but the impact of complexity remains relatively unexplored. |
| Approach: | They propose to add a specified number of nodes to instructions’ semantic trees to enhance the instruction complexity in a controllable manner. |
| Outcome: | The proposed approach outperforms diverse yet complex instructions under the same token budget and can control the difficulty level of modified instructions. |