Papers by Yujia Qin
Recyclable Tuning for Continual Pre-training (2023.findings-acl)
Copied to clipboard
Yujia Qin, Cheng Qian, Xu Han, Yankai Lin, Huadong Wang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou
| Challenge: | Continual pre-training is the paradigm where pre-trained language models acquire fresh knowledge and gradually get upgraded. |
| Approach: | They propose to use adapted weights to recycle old PLMs for continual pre-training . they propose to combine initialization and distillation methods to achieve better performance . |
| Outcome: | The proposed method improves the convergence and performance of the upgraded PLM. |
RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation (2024.emnlp-demo)
Copied to clipboard
Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, Xiaoyin Che, Zhiyuan Liu, Maosong Sun
| Challenge: | Xia et al., 2018) demonstrate that a large language model can generate and maintain high-quality code documentation. |
| Approach: | They propose a large language model powered open-source framework for generating, maintaining, and updating code documentation. |
| Outcome: | The proposed framework generates high-quality documentation for the entire project. |
Exploring Mode Connectivity for Pre-trained Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent years have witnessed the prevalent application of pre-trained language models (PLMs) in NLP. From the perspective of parameter space, PLMs provide generic initialization, starting from which high-performance minima could be found. |
| Approach: | They investigate the geometric connections of different minima through the lens of mode connectivity, which measures whether two minima can be connected with a low-loss path. |
| Outcome: | The proposed model can be used to find low-loss paths between two minima, and to understand how their mode connectivity affects their task knowledge. |
Knowledge Inheritance for Pre-trained Language Models (2022.naacl-main)
Copied to clipboard
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou
| Challenge: | Existing large-scale pre-trained language models are mainly trained from scratch individually, ignoring that many well-taught PLMs are available. |
| Approach: | They propose a pre-training framework called knowledge inheritance and propose auxiliary supervision to efficiently learn larger PLMs. |
| Outcome: | The proposed framework can be used to train large-scale language models with huge parameters and a large dataset can be adapted to domain adaptation and knowledge transfer. |
Pass off Fish Eyes for Pearls: Attacking Model Selection of Pre-trained Models (2022.acl-long)
Copied to clipboard
| Challenge: | Existing feature-based model selection methods are vulnerable to fine-tuning and are not reliable indicators for the PTM’s transferability. |
| Approach: | They propose feature-based model selection methods which assess PTMs’ transferability to a specific task in a fast way without fine-tuning. |
| Outcome: | The proposed methods can make FMS mistakenly judge transferability of models and can be combined with the backdoor attack to misguide the FMS to select poisoned models. |
ProQA: Structural Prompt-based Pre-training for Unified Question Answering (2022.naacl-main)
Copied to clipboard
Wanjun Zhong, Yifan Gao, Ning Ding, Yujia Qin, Zhiyuan Liu, Ming Zhou, Jiahai Wang, Jian Yin, Nan Duan
| Challenge: | Existing QA research on question answering is focused on specific question types, knowledge domains, or reasoning skills. |
| Approach: | They propose a unified QA paradigm that solves various tasks through a single model. |
| Outcome: | The proposed model improves QA-centric ability on 11 QA benchmarks. |
Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents (2024.acl-long)
Copied to clipboard
Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, Maosong Sun
| Challenge: | Current language model-driven agents lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions. |
| Approach: | They propose a benchmark to inspect users’ implicit intentions through explicit queries and a model expert as the upstream in agent design to enhance user-agent interaction. |
| Outcome: | The proposed approach excels at identifying vague user tasks, recovering and summarizing critical missing information, setting precise and necessary agent execution goals, and minimizing redundant tool usage, thus boosting overall efficiency. |
Large Language Model-based Human-Agent Collaboration for Complex Task Solving (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to the development of LLM-based autonomous agents. |
| Approach: | They propose a Reinforcement Learning-based Human-Agent Collaboration method which trains a policy model to determine the most opportune stages for human intervention within the task-solving process. |
| Outcome: | The proposed method improves human-agent collaboration significantly through well-planned, limited human intervention. |
Exploring the Impact of Model Scaling on Parameter-Efficient Tuning (2023.emnlp-main)
Copied to clipboard
Yusheng Su, Chi-Min Chan, Jiali Cheng, Yujia Qin, Yankai Lin, Shengding Hu, Zonghan Yang, Ning Ding, Xingzhi Sun, Guotong Xie, Zhiyuan Liu, Maosong Sun
| Challenge: | Parameter-efficient tuning (PET) methods can drive large pre-trained language models by training only minimal parameters. |
| Approach: | They propose a parameter-efficient tuning method that is compatible with a tunable module and uses a random number generator to optimize fewer table parameters. |
| Outcome: | The proposed method is compatible with a tunable module and tested on 11 NLP tasks. |
Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies show that large pre-trained language models can be adapted to particular tasks in a parameter-efficient manner. |
| Approach: | They propose to use lightweight parameters to transfer them between tasks to obtain similarity between tasks. |
| Outcome: | The proposed methods show an improvement of 5%8% over baselines and could largely facilitate task-level knowledge transfer. |
On Transferability of Prompt Tuning for Natural Language Processing (2022.naacl-main)
Copied to clipboard
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, Jie Zhou
| Challenge: | Pre-trained language models (PLMs) can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but require much more training time than fine-timing. |
| Approach: | They empirically investigate the transferability of soft prompts across different downstream tasks and PLMs to determine what decides prompt transferability. |
| Outcome: | The proposed method can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but requires much more training time than fine-timing. |
FPT: Improving Prompt Tuning Efficiency via Progressive Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent prompt tuning (PT) has gained increasing attention as a parameter-efficient way of tuning pre-trained language models (PLMs). |
| Approach: | They propose a prompt tuning algorithm that uses a small-scale partial PLM and progressively expands its depth and width until the full-model size. |
| Outcome: | The proposed method could save over 30% of training computations while achieving comparable performance. |
GUICourse: From General Vision Language Model to Versatile GUI Agent (2025.acl-long)
Copied to clipboard
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, Maosong Sun
| Challenge: | Graphical User Interfaces (GUIs) are a pivotal medium for human-computer interaction. |
| Approach: | They propose a series of datasets for training visual-based GUI agents using general VLMs. |
| Outcome: | The proposed GUICourse datasets show that even a small-sized GUI agent performs better on GUI tasks. |
StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models (2024.findings-acl)
Copied to clipboard
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, Yang Liu
| Challenge: | Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning. |
| Approach: | They propose a virtual API server and stable evaluation system to assess the stability of large-scale real-time APIs. |
| Outcome: | The proposed benchmarks demonstrate the stability of the proposed system and its caching system. |
WebCPM: Interactive Web Search for Chinese Long-form Question Answering (2023.acl-long)
Copied to clipboard
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, Jie Zhou
| Challenge: | Long-form question answering requires two procedures: information retrieval and information synthesis. |
| Approach: | They propose a Chinese long-form question answering dataset called WebCPM . the dataset is based on a web search interface that engages with a search engine in real time . |
| Outcome: | The proposed dataset generates answers that are no worse than human-written ones . the dataset is the first Chinese LFQA dataset . |
bert2BERT: Towards Reusable Pretrained Language Models (2022.acl-long)
Copied to clipboard
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, Qun Liu
| Challenge: | Pre-training large language models can be expensive and wasteful. |
| Approach: | They propose a method which can transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and a two-stage learning method to further accelerate the pre-training. |
| Outcome: | The proposed method can transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and significantly improve the pre-training efficiency of the large model. |
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (2021.acl-long)
Copied to clipboard
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, Jie Zhou
| Challenge: | Existing pre-training objectives do not explicitly model relational facts in text . Experimental results show that ERICA can improve typical PLMs on several language understanding tasks, including relation extraction, entity typing and question answering. |
| Approach: | They propose a contrastive learning framework ERICA to obtain a deep understanding of entities and relations in text. |
| Outcome: | The proposed framework can improve PLMs on several language understanding tasks, especially under low-resource settings. |
DebugBench: Evaluating Debugging Capability of Large Language Models (2024.findings-acl)
Copied to clipboard
Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, Maosong Sun
| Challenge: | Large language models (LLMs) have demonstrated exceptional coding capabilities, but their debugging capabilities remain relatively unexplored. |
| Approach: | They propose a debugging benchmark consisting of 4,253 LLMs with four major bug categories and 18 minor types in C++, Java, and Python. |
| Outcome: | The proposed benchmark covers four major bug categories and 18 minor types in C++, Java, and Python. |
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study validates the effectiveness of chat language models by fine-tuning instruction data. |
| Approach: | They propose to use a large-scale dataset of instructional conversations to fine-tune a conversational model on instruction data. |
| Outcome: | The proposed model outperforms open-source models in key metrics including scale, average length, diversity, coherence, etc. |
Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Parameter-Efficient Tuning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing delta tuning algorithms freeze most of the parameters and only optimize minimal adaptive parameters. |
| Approach: | They propose to decompose DETs into a unified optimization subspace and conduct optimization within the subspace. |
| Outcome: | The proposed DETs achieve comparable performance to the original DET and can be transferred to another DET with non-trivial performance. |
ELLE: Efficient Lifelong Pre-training for Emerging Data (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained language models are typically trained with static data, ignoring that streaming data of various sources may continuously grow. |
| Approach: | They propose to use function preserved model expansion to expand existing PLM's width and depth to improve efficiency of knowledge acquisition. |
| Outcome: | The proposed model improves pre-training efficiency and performance over existing models on BERT and GPT. |
Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub (2025.acl-long)
Copied to clipboard
Bohan Lyu, Xin Cong, Heyang Yu, Pan Yang, Cheng Qian, Zihe Wang, Yujia Qin, Yining Ye, Yaxi Lu, Chen Qian, Zhong Zhang, Yukun Yan, Yankai Lin, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing approaches lack flexibility to address diverse and ever-evolving user queries in open domains. |
| Approach: | They propose to evaluate LLMs on open-domain knowledge that requires tools to solve diverse and ever-evolving user queries. |
| Outcome: | The proposed system outperforms baselines in the open domain task-solving benchmark. |
CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning. |
| Approach: | They propose a framework that enables LLMs to create their own tools using documentation and code realization. |
| Outcome: | The proposed framework outperforms existing chain-of-thought, program-of thought, and tool-using baselines on MATH and TabMWP benchmarks. |