Papers by Canwen Xu
LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval (2022.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that LaPraDoR is state-of-the-art compared with supervised dense retrieval models. |
| Approach: | They propose a pretrained dual-tower dense retriever that does not require supervised data for training. |
| Outcome: | The proposed method achieves state-of-the-art performance on 18 datasets of 9 zero-shot text retrieval tasks. |
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022.acl-demo)
Copied to clipboard
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, Alexander Rush
| Challenge: | PromptSource is a system for creating, sharing, and using natural language prompts . prompts are used to train and query language models in zero-shot learning settings . |
| Approach: | PromptSource is a system for creating, sharing, and using natural language prompts . et al.: using prompts to train and query language models is emerging area in NLP . they propose a templating language for defining data-linked prompts, a user interface that iterates on prompt development . |
| Outcome: | PromptSource is a system for creating, sharing, and using natural language prompts . it has a templating language for defining data-linked prompts and a community-driven set of guidelines . |
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies on compression of pretrained language models usually use preserved accuracy as the metric for evaluation. |
| Approach: | They propose two new metrics that measure how closely a compressed model mimics the original model. |
| Outcome: | The proposed metrics measure how closely a compressed model (i.e., student) mimics the original model (e.g., teacher). |
Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Despite the promising potential of chat models, they are only accessible through restricted APIs, creating barriers for new research and progress in the field. |
| Approach: | They propose a pipeline that can automatically generate a high-quality multi-turn chat corpus by leveraging ChatGPT to engage in a conversation with itself. |
| Outcome: | The proposed pipeline generates a high-quality multi-turn chat corpus by leveraging ChatGPT to engage in a conversation with itself, simulating both user and AI responses. |
Datasets: A Community Library for Natural Language Processing (2021.emnlp-demo)
Copied to clipboard
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, Thomas Wolf
| Challenge: | Contemporary NLP systems use many different datasets at significantly varying scale and level of annotation. |
| Approach: | a community library for contemporary NLP is available at https://github.com/datasets . the library includes more than 650 unique datasets and has more than 250 contributors a year after its initial development . |
| Outcome: | the library includes more than 650 unique datasets and has more than 250 contributors . it supports a variety of cross-dataset research projects and shared tasks . |
Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge (2021.naacl-main)
Copied to clipboard
| Challenge: | Cant is important for understanding advertising, comedies and dogwhistle politics . currently, there are very few resources available for the research of cant . |
| Approach: | They propose a large and diverse dataset for creating and understanding cant from a computational linguistics perspective. |
| Outcome: | The proposed dataset can be used to test word embedding similarity and pretrained language models. |
BERT Learns to Teach: Knowledge Distillation with Meta Learning (2022.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods are based on teacher model, but have drawbacks . a teacher model is fixed during training, but meta learning can improve student performance . |
| Approach: | They propose a meta learning framework that allows the teacher network to learn to better transfer knowledge to the student network. |
| Outcome: | Experiments show that MetaDistil can improve on existing methods and is less sensitive to student capacity and hyperparameters. |
Spoiler Detection as Semantic Text Matching (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on spoiler detection shows promising results in safeguarding viewers from general spoilers, but it fails to address the issue of users abstaining from show-related content during their watch. |
| Approach: | They propose to use semantic text matching to assign an episode number to a spoiler given a specific TV show and a dataset to evaluate its performance. |
| Outcome: | The proposed dataset can be used to evaluate the performance of the proposed model and to compare it with other datasets. |
A Survey on Dynamic Neural Networks for Natural Language Processing (2023.findings-eacl)
Copied to clipboard
| Challenge: | Dynamic neural networks can scale up pretrainable models with sub-linear increases in computation and time. |
| Approach: | They summarize the progress of three types of dynamic neural networks in NLP . skimming, mixtures of experts, and early exit are among the most popular . |
| Outcome: | The proposed models can scale up with sub-linear increases in computation and time . skimming, mixture of experts, and early exit are the most popular approaches . |
Transformers: State-of-the-Art Natural Language Processing (2020.emnlp-demos)
Copied to clipboard
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander Rush
| Challenge: | Transformers is an open-source library that aims to open up advances in natural language processing to the wider machine learning community. |
| Approach: | they propose an open-source library that aims to open up advances in machine learning to the wider community. |
| Outcome: | Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. |
Efficiently Tuned Parameters Are Task Embeddings (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for intermediate-task transfer are computationally infeasible to experiment with all intermediate combinations. |
| Approach: | They propose to use task-specific parameters updated in parameter-efficient tuning methods to predict inter-task transferability. |
| Outcome: | The proposed approach outperforms existing methods while being conceptually simple and computationally efficient. |
InforMask: Unsupervised Informative Masking for Language Model Pretraining (2022.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling is used for pretraining large language models for knowledge-intensive tasks. |
| Approach: | They propose an unsupervised masking strategy that exploits Pointwise Mutual Information to select the most informative tokens to mask. |
| Outcome: | The proposed strategy outperforms random masking and previously proposed masking strategies on the factual recall benchmark LAMA and the question answering benchmark SQuAD v1 and v2. |
Optimizing Reasoning for Text-to-SQL with Execution Feedback (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models excel in many reasoning tasks, but their ability to leverage Chain-of-Thought (CoT) reasoning remains underexplored. |
| Approach: | They propose a framework that iteratively optimizes open-source LLMs by combining CoT reasoning with off-policy and on-poly DPO, relying solely on execution accuracy as feedback. |
| Outcome: | The proposed framework improves execution accuracy on BIRD and Spider datasets. |
Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot Classification (2022.naacl-main)
Copied to clipboard
| Challenge: | Prompt-based learning is an emerging paradigm for exploiting knowledge learned by a pretrained language model. |
| Approach: | They propose a method to automatically select label mappings for few-shot text classification with prompting. |
| Outcome: | The proposed method achieves competitive performance on the GLUE benchmark without human effort or external resources. |
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020.emnlp-main)
Copied to clipboard
| Challenge: | a novel approach to compress neural networks by progressive module replacement is proposed . a number of techniques have been proposed to compress pretraining and fine-tuning models . |
| Approach: | They propose a model compression approach that divides BERT into modules and builds their compact substitutes. |
| Outcome: | The proposed approach outperforms existing knowledge distillation approaches on GLUE benchmark . it is based on a model that divides the original BERT into several modules and builds their substitutes . |
Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing text infilling objectives for pretrained language models require self-supervision by masking out tokens or spans in text. |
| Approach: | They propose to extend text infilling to a self-supervised sequence-to-sequence (Seq2Sequen) task. |
| Outcome: | The proposed task improves the model's performance on various natural language generation tasks. |
Automatic Pair Construction for Contrastive Post-training (2024.findings-naacl)
Copied to clipboard
Canwen Xu, Corby Rosset, Ethan Chau, Luciano Corro, Shweti Mahajan, Julian McAuley, Jennifer Neville, Ahmed Awadallah, Nikhil Rao
| Challenge: | Large language models (LLMs) have unprecedented proficiency in a wide array of tasks. |
| Approach: | They propose a way to construct contrastive data using preference pairs from multiple models of varying strengths using SLiC and DPO. |
| Outcome: | The proposed method outperforms existing models like Orca in the comparison of SLiC and DPO with SFT baselines. |
MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing datasets labeled for one task hinder multi-task learning . task-specific data make models learn task-related leakage features rather than meaningful knowledge that could generalize to other tasks. |
| Approach: | They propose to jointly label large-scale NLP dataset MATINF . it contains 1.07 million question-answer pairs with human-labeled categories . |
| Outcome: | The proposed dataset is applicable for classification, question answering, and summarization. |
Small Models are Valuable Plug-ins for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are difficult to fine-tune due to their huge weights and limited context length. |
| Approach: | They propose an approach which allows black-box LLMs to work with locally fine-tuned smaller models, resulting in superior performance on supervised tasks. |
| Outcome: | The proposed approach overcomes the challenges of poor performance and instability of In-Context Learning (ICL) while reducing the complexity of in-context learning. |
UnihanLM: Coarse-to-Fine Chinese-Japanese Language Model Pretraining with the Unihan Database (2020.aacl-main)
Copied to clipboard
| Challenge: | Chinese and Japanese share many characters with similar surface morphology. |
| Approach: | They propose a Chinese-Japanese pretrained masked language model with a coarse-to-fine training approach to exploit the shared knowledge across the languages. |
| Outcome: | The proposed model is effective on mono- and cross-lingual Chinese and Japanese tasks. |
Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders (2020.acl-main)
Copied to clipboard
| Challenge: | Existing conditional generation models cannot handle emerging conditions due to their joint end-to-end learning fashion. |
| Approach: | They propose a framework for conditional text generation that decouples the text generation module from the condition representation module to allow "one-to-many" conditional generation. |
| Outcome: | The proposed framework decouples the text generation module from the condition representation module to allow “one-to-many” conditional generation. |