Papers by Tao Kong
UNComp: Can Matrix Entropy Uncover Sparsity? — A Compressor Design from an Uncertainty-Aware Perspective (2025.emnlp-main)
Copied to clipboard
Jing Xiong, Jianghan Shen, Fanghua Ye, Chaofan Tao, Zhongwei Wan, Jianqiao Lu, Xun Wu, Chuanyang Zheng, Zhijiang Guo, Min Yang, Lingpeng Kong, Ngai Wong
| Challenge: | Deploying large language models (LLMs) for long-context inference remains challenging due to their substantial memory and computational demands. |
| Approach: | They propose an uncertainty-aware framework that leverages truncated matrix entropy to identify areas of low information content. |
| Outcome: | The proposed framework reduces the KV cache size to 4.74% of the original and achieves a 6% speedup. |
ProcWorld: Benchmarking Large Model Planning in Reachability-Constrained Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for embodied spatial reasoning and long-term planning are non-trivial due to the combinatorial complexity of long-horizon abstract reasoning. |
| Approach: | They propose a large-scale benchmark for partially observable embodied spatial reasoning and long-term planning with large language models and vision language models. |
| Outcome: | The proposed model performs better in 16 task types, 5,000 rooms, and over 10 million evaluation trajectories with diverse data distribution. |
What Matters in Training a GPT4-Style Language Model with Multimodal Inputs? (2024.naacl-long)
Copied to clipboard
Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, Tao Kong, Ruihua Song
| Challenge: | Recent advances in GPT-4V have demonstrated remarkable multi-modal capabilities in processing image inputs and following open-ended instructions. |
| Approach: | They propose a plug-and-play technique to enhance multi-modal LLMs . they propose 'lynx' to train multi-modal LLM models . |
| Outcome: | The proposed training strategy improves understanding accuracy and instruction-following proficiency of multi-modal models. |
Towards Unifying Reference Expression Generation and Comprehension (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for REG and REC have distinct inputs and connections between them . a new model for REg and reprehension is needed to solve these problems . |
| Approach: | They propose a unified model for REG and REC that fuses image, region and text . they propose Vision-conditioned Masked Language Modeling and Text-Conditioned Region Prediction . |
| Outcome: | The proposed model outperforms existing models on REG and REC tasks. |
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate training data with pre-trained language models have been found effective in various scenarios. |
| Approach: | They propose an unsupervised zero-shot learning method that generates a dataset from scratch and trains a tiny task model under supervision of the synthesized dataset. |
| Outcome: | The proposed method is annotated-free and efficient, but can provide useful insights from the perspective of data-free model-agnostic knowledge distillation and unreferenced text generation evaluation. |
Generating Data for Symbolic Language with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) bring performance and complexity, but they incur a large computational cost in practice. |
| Approach: | They propose a task-based model which uses large language models to generate symbolic language data by an informative prompt and agreement-based verifier. |
| Outcome: | The proposed model can generate symbolic language data with a few human demonstrations and saves a considerable amount of inference effort. |
Bilateral Masking with prompt for Knowledge Graph Completion (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing word matching methods fail to obtain satisfactory single embedding representations for entities. |
| Approach: | They propose a bi-encoder-based approach to enhance entity representations by using prompts to narrow the distance between the predicted entity and the known entity. |
| Outcome: | The proposed model achieves state-of-the-art performance on the WN18RR dataset. |
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022.emnlp-main)
Copied to clipboard
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, Tao Yu
| Challenge: | Structured knowledge grounding (SKG) uses structured knowledge to complete user requests . since inputs and outputs of SKG tasks are heterogeneous, they have been studied separately . |
| Approach: | They propose a framework that unifies 21 SKG tasks into a text-to-text format . they use unifiedSKG to benchmark T5 with different sizes . |
| Outcome: | The proposed framework unifies 21 SKG tasks into a text-to-text format . it achieves state-of-the-art performance on almost all of the 21 tasks, the authors show . |
MTO: A Multi-turn Conversational Text-to-OverpassQL Dataset for Enhanced OpenStreetMap Query Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | a framework for constructing multi-turn Text-to-OverpassQL dialogue datasets is proposed . a dataset of over 7,800 dialogues contains more than 20,000 individual utterances . |
| Approach: | They propose a framework for constructing multi-turn Text-to-OverpassQL dialogue datasets . they convert Overpass queries into syntax trees using a custom parser based on OverpassQl . |
| Outcome: | The proposed dataset includes over 7,800 dialogues, each containing 2 to 4 user utterances . it is the first multi-turn Text-to-OverpassQL dataset built upon the OverpassNL corpus . |
Textual Dataset Distillation via Language Model Embedding (2024.findings-emnlp)
Copied to clipboard
| Challenge: | prevailing methods for dataset distillation generate distilled data as embedding vectors, which are not human-readable. |
| Approach: | They propose a model-agnostic, data-efficient method that leverages Language Model embeddings . their method offers enhanced flexibility and improved transferability . |
| Outcome: | The proposed method achieves comparable performance with faster processing times compared to other methods . it offers enhanced flexibility and improved transferability, expanding the range of potential applications . |
ImgTrojan: Jailbreaking Vision-Language Models with ONE Image (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on the safety of large language models (LLMs) with human values have focused on the integration of multi-modal user input into these models. |
| Approach: | They propose a method to bypass safety constraints of large language models by using poisoned images instead of original textual captions. |
| Outcome: | The proposed attack bypasses safety constraints of large language models (VLMs) by replacing the original textual captions with malicious jailbreak prompts. |
BPID: A Benchmark for Personal Identity Deduplication (2024.emnlp-industry)
Copied to clipboard
Runhui Wang, Yefan Tao, Adit Krishnan, Luyang Kong, Xuanqing Liu, Yuqian Deng, Yunzhao Yang, Henrik Johnson, Andrew Borthwick, Shobhit Gupta, Aditi Gundlapalli, Davor Golac
| Challenge: | Data deduplication is a critical task in data management and mining, focused on consolidating duplicate records that refer to the same entity. |
| Approach: | They propose to use a dataset with 1,000,000 unlabeled synthetic PII profiles and a subset of 10,000 pairs curated and labeled as matches or non-matches. |
| Outcome: | The proposed datasets contain synthetic profiles built from publicly available sources that do not represent real individuals. |
ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on dataset-generation-based zero-shot learning has shown promising results by training a task-specific model with a dataset synthesized from large pre-trained language models (PLMs). |
| Approach: | They propose a progressive zero-shot dataset generation framework which leverages feedback from the task-specific model to guide the generation of new training data via in-context examples. |
| Outcome: | The proposed framework achieves on-par or superior performance with only 1% synthetic dataset size, when compared to baseline methods without in-context feedback. |