Papers by Daixuan Cheng
Snapshot-Guided Domain Adaptation for ELECTRA (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain-specific knowledge of domain-related tasks is lacking in pre-trained language models. |
| Approach: | They propose a domain-adaptation method which can dynamically select domain-specific tokens and guide the discriminator to emphasize them, without introducing new training parameters. |
| Outcome: | The proposed method can capture domain-specific knowledge of domain-related tasks without introducing new training parameters. |
MDR: Model-Specific Demonstration Retrieval at Inference Time for In-Context Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for retrieval-based in-context learning ignore model biases and fail to retrieve the most appropriate demonstrations for different LLMs. |
| Approach: | They propose a model-specific demonstration retrieval method that considers the biases of different LLMs at inference time. |
| Outcome: | The proposed method improves performance on seen and unseen tasks with multi-scale inference LLMs by up to 41.2%. |
Instruction Pre-Training: Language Models are Supervised Multitask Learners (2024.emnlp-main)
Copied to clipboard
| Challenge: | Unsupervised multitask pre-training has been the key to the success of language models (LMs) however, scaling it in the post-training stage trends towards better generalization. |
| Approach: | They propose a framework that augments massive raw corpora with instruction-response pairs to pre-train LMs. |
| Outcome: | The proposed framework augments massive raw corpora with instruction-response pairs to pre-train LMs. |
On Domain-Adaptive Post-Training for Multimodal Large Language Models (2025.findings-emnlp)
Copied to clipboard
Daixuan Cheng, Shaohan Huang, Ziyu Zhu, Xintong Zhang, Xin Zhao, Zhongzhi Luan, Bo Dai, Zhenliang Zhang
| Challenge: | Adapting general multimodal large language models to specific domains is important for practical applications. |
| Approach: | They investigate domain adaptation of multimodal large language models via post-training . they develop a generate-then-filter pipeline that curates diverse visual instruction tasks . |
| Outcome: | The proposed model outperforms existing models in domain adaptation by combining data from open-source models with training pipelines. |
UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation (2023.emnlp-main)
Copied to clipboard
Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Weiwei Deng, Qi Zhang
| Challenge: | Large Language Models (LLMs) have impressive capabilities but need for task-specific prompt engineering can hinder their generalization. |
| Approach: | They propose a lightweight and versatile retriever that automatically retrieves prompts for a given zero-shot task input. |
| Outcome: | The proposed model is universally applicable across tasks and models . it mitigates hallucination problem in chatGPT, and it improves even the strongest LLMs. |