Papers by Pengtao Xie
An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to learning data representations using contrastive learning perform data augmentation and contrastive training separately. |
| Approach: | They propose a framework that performs data augmentation and contrastive learning end-to-end . they propose to combine data augmented with text encoders to optimize for contrastive training . |
| Outcome: | Experiments on GLUE and Gururangan datasets show the proposed framework is effective in NLP. |
Self-supervised Regularization for Text Classification (2021.tacl-1)
Copied to clipboard
| Challenge: | Text classification models are prone to overfitting when limited texts are available for training. |
| Approach: | They propose a data-dependent regularization approach based on self-supervised learning . they define auxiliary tasks on input data without using human-provided labels . |
| Outcome: | Experiments on 17 text classification datasets demonstrate the effectiveness of the proposed method. |
AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Large-scale pretraining followed by task-specific finetuning has achieved great success in various NLP tasks. |
| Approach: | They propose a meta learning based framework for automatically identifying the optimal rank of each LoRA layer. |
| Outcome: | The proposed framework is based on a meta learning based framework that can identify the optimal rank of each LoRA layer. |
A Neural Architecture for Automated ICD Coding (P18-1)
Copied to clipboard
| Challenge: | Medical coding is time-consuming, expensive, and error prone. |
| Approach: | They propose to use diagnosis descriptions (DDs) of a patient as inputs to select the most relevant ICD codes. |
| Outcome: | The proposed algorithms perform on a clinical dataset with 59K patient visits. |
A Multi-Level Optimization Framework for End-to-End Text Augmentation (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for text augmentation perform data augmentation and downstream tasks separately. |
| Approach: | They propose a framework to perform text augmentation and the downstream task end-to-end. |
| Outcome: | The proposed framework performs text augmentation and the downstream task end-to-end on a text classification dataset. |
Towards Visual Question Answering on Pathology Images (2021.acl-short)
Copied to clipboard
| Challenge: | Pathology imaging is used for identifying the causes and effects of diseases or injuries. |
| Approach: | They propose a pathological visual question answering framework to analyze pathology images and answer medical questions related to these images. |
| Outcome: | The proposed framework performs self-supervised pretraining and finetuning end-to-end to learn powerful visual and textual representations jointly and automatically identifies and excludes noisy self-controlled examples from pretraining. |
Can Prompts Rewind Time for LLMs? Evaluating the Effectiveness of Prompted Knowledge Cutoffs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely used for temporal prediction tasks . however, their reliance on pretraining data can lead to contamination concerns . |
| Approach: | They investigate the capability of prompting to simulate an earlier knowledge cutoff in large language models. |
| Outcome: | The proposed model fails to induce forgetting when the forgotten content is not directly asked but causally related to the query. |
MedDialog: Large-scale Medical Dialogue Datasets (2020.emnlp-main)
Copied to clipboard
Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, Hongchao Fang, Penghui Zhu, Shu Chen, Pengtao Xie
| Challenge: | telemedicine is a medical practice that provides patient care remotely using video conferencing tools. |
| Approach: | They build large-scale medical dialogue datasets to facilitate research . they pretrain several models on the Chinese MedDialog dataset and compare their performance . |
| Outcome: | The proposed datasets show that models trained on MedDialog can generate doctor-like medical dialogues. |
Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Instruction-fine-tuned large language models (LLMs) under 14B parameters underperform on NLU tasks . we explore a framework to improve the NLU capabilities of LLMs . |
| Approach: | They propose to use Proximal Policy Optimization to improve NLU capabilities . they frame NLU as a reinforcement learning environment and optimize for reward signals . |
| Outcome: | The proposed framework outperforms supervised fine-tuning on GLUE and superGLUE tasks. |
BioTool: A Comprehensive Tool-Calling Dataset for Enhancing Biomedical Capabilities of Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | despite the success of large language models, their performance in highly specialized domains remains unsatisfactory. |
| Approach: | They propose a biomedical tool-calling dataset designed for fine-tuning LLMs . the dataset contains 34 frequently used tools from the NCBI, Ensembl, and UniProt databases . |
| Outcome: | The proposed dataset outperforms commercial LLMs on biomedical domains. |
On the Generation of Medical Dialogs for COVID-19 (2021.acl-short)
Copied to clipboard
Meng Zhou, Zechen Li, Bowen Tan, Guangtao Zeng, Wenmian Yang, Xuehai He, Zeqian Ju, Subrato Chakravorty, Shu Chen, Xingyi Yang, Yichen Zhang, Qingyang Wu, Zhou Yu, Kun Xu, Eric Xing, Pengtao Xie
| Challenge: | under the pandemic of COVID-19, people experiencing COVI D19-related symptoms have a pressing need to consult doctors. |
| Approach: | They develop a medical dialog system that can provide COVID19-related consultations . they use two dialog datasets containing conversations between doctors and patients . |
| Outcome: | The proposed system can provide COVID19-related consultations, but is too small compared with general-domain dialog datasets. |
On the Automatic Generation of Medical Imaging Reports (P18-1)
Copied to clipboard
| Challenge: | a complete medical imaging report contains multiple heterogeneous forms of information, including findings and tags . abnormal regions in medical images are difficult to identify and the reports are typically long, containing multiple sentences. |
| Approach: | They propose a multi-task learning framework which predicts tags and generates paragraphs for abnormal regions in medical images. |
| Outcome: | The proposed framework can generate long paragraphs on two publicly available datasets. |
Defense against Prompt Injection Attacks via Mixture of Encodings (2025.naacl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have emerged as a dominant approach for a wide range of NLP tasks, but external content embeds malicious instructions that manipulate the LLM’s output. |
| Approach: | They propose a mixture of encodings defense mechanism which utilizes multiple character encodes to degrade LLM performance on certain NLP tasks. |
| Outcome: | The proposed method achieves one of the lowest attack success rates under prompt injection attacks while maintaining high performance across all NLP tasks. |
Structured Self-Supervised Pretraining for Commonsense Knowledge Graph Completion (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches focus on generating concepts that have direct and obvious relationships with existing concepts and lack an ability to generate unobvious concepts. |
| Approach: | They propose a general graph-to-paths pretraining framework that leverages high-order structures in CKGs to capture high-level relationships between concepts. |
| Outcome: | The proposed framework can capture high-order relationships between concepts in four special cases: long path, path-to-path, router, and graph-node-path. |
MetaWeighting: Learning to Weight Tasks in Multi-Task Learning (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing task weighting methods assign weights only based on training losses, while ignoring the gap between the training loss and generalization loss. |
| Approach: | They propose a task weighting algorithm which automatically weights the tasks via a learning-to-learn paradigm and a multi-task text classification paradigm. |
| Outcome: | Extensive experiments show that the proposed method outperforms existing methods in multi-task text classification. |
Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts (2024.naacl-long)
Copied to clipboard
| Challenge: | Pretrained language models have advanced natural language processing tasks significantly, but finetuning them on low-resource datasets presents significant challenges such as instability and overfitting. |
| Approach: | They propose a regularization method based on attention-guided weight mixup for finetuning PLMs on low-resource datasets. |
| Outcome: | The proposed method improves generalization and combats overfitting on two splits of the training dataset. |