Papers by Mengzhou Xia
Predicting Performance for Natural Language Processing Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language processing (NLP) is a vast field, with a wide variety of tasks, languages, and domains. |
| Approach: | They build regression models to predict evaluation score of an NLP experiment . they find that their models can produce meaningful predictions over unseen languages . |
| Outcome: | The proposed model outperforms baseline models and human experts on 9 different tasks. |
Structured Pruning Learns Compact and Accurate Models (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have high costs in terms of storage, memory, and computation time. |
| Approach: | They propose a task-specific structured pruning method CoFi which provides highly parallelizable subnetworks and matches distillation methods in both accuracy and latency. |
| Outcome: | The proposed method matches the distillation methods in accuracy and latency without resorting to unlabeled data. |
Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained masked language models perform few-shot learning, but discriminative models like ELECTRA do not fit into the paradigm. |
| Approach: | They propose to use ELECTRA to train pre-trained models to score originality of target options without introducing new parameters. |
| Outcome: | The proposed model outperforms masked language models in a wide range of tasks without adding new parameters. |
Non-Parametric Few-Shot Learning for Word Sense Disambiguation (2021.naacl-main)
Copied to clipboard
| Challenge: | Word sense disambiguation (WSD) is a problem in natural language processing . 84% of annotated words have less than 10 examples in the long-tail distribution . |
| Approach: | They propose a non-parametric few-shot learning approach to mitigate word sense disambiguation . they use a metric space to compute distances among the senses of a given word . |
| Outcome: | The proposed method achieves a 75.1 F1 score on the unified evaluation benchmark. |
Don’t Prompt, Search! Mining-based Zero-Shot Learning with Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has obtained strong zero-shot results by prompting language models. |
| Approach: | They propose a mining-based approach that uses regular expressions to mine labeled examples from unlabeled corpora and fine tune a pretrained model. |
| Outcome: | The proposed method outperforms prompting on a wide range of tasks when using comparable templates. |
Generalized Data Augmentation for Low-Resource Translation (P19-1)
Copied to clipboard
| Challenge: | Low-resource language pairs with a lack of parallel data pose challenges for machine translation . data augmentation using monolingual data is an effective way to alleviate the problem . |
| Approach: | They propose a general framework for data augmentation for low-resource machine translation using monolingual data and a related high-resourced language. |
| Outcome: | The proposed method improves translation quality by 1.5 to 8 BLEU points under extreme low-resource settings compared to baselines. |
Domain Adaptation of Neural Machine Translation by Lexicon Induction (P19-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) is sensitive to domain shift, resulting in failure for sentences with large numbers of unknown words and lack of supervision for domain-specific words. |
| Approach: | They propose an unsupervised method which fine-tunes a pre-trained out-of-domain NMT model using a pseudo-in-domain corpus. |
| Outcome: | The proposed method improves in five domains without using in-domain parallel sentences and up to 2 BLEU over strong back-translation baselines. |
InstructEval: Systematic Evaluation of Instruction Selection Methods (2024.findings-naacl)
Copied to clipboard
| Challenge: | In-context learning (ICL) performs tasks by prompting a large language model using an instruction and a small set of annotated examples. |
| Approach: | They develop an ICL evaluation suite to evaluate the performance of popular instruction selection methods. |
| Outcome: | The proposed evaluation suite compares instruction selection methods over five metrics relevant to ICL. |
MABEL: Attenuating Gender Bias using Textual Entailment Data (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for mitigating gender bias in language models are insufficient or inconsistent. |
| Approach: | They propose a method for attenuating gender bias using entailment labels . they use a contrastive learning objective on counterfactually augmented enanglement pairs . |
| Outcome: | The proposed method outperforms previous task-agnostic debiasing approaches on intrinsic and extrinsic metrics and preserves task performance after fine-tuning on downstream tasks. |
LitSearch: A Retrieval Benchmark for Scientific Literature Search (2024.emnlp-main)
Copied to clipboard
| Challenge: | Literature search questions pose significant challenges for modern retrieval systems . a lack of domain expertise and reasoning through lengthy papers is a challenge . |
| Approach: | They propose a retrieval benchmark for literature search queries using inline citations from papers and questions about recently published papers. |
| Outcome: | The proposed retrieval benchmarks outperform state-of-the-art retrieval models and reranking pipelines. |
Choosing Transfer Languages for Cross-Lingual Learning (P19-1)
Copied to clipboard
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, Graham Neubig
| Challenge: | Cross-lingual transfer is a useful tool for improving performance of natural language processing (NLP) on low-resource languages. |
| Approach: | They propose to use cross-lingual transfer to improve accuracy of low-resource languages . they build models that consider features to perform prediction on such languages based on ranking problem . |
| Outcome: | The proposed model predicts good transfer languages much better than baselines considering single features in isolation. |
Training Trajectories of Language Models Across Scales (2023.acl-long)
Copied to clipboard
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. |
| Approach: | They analyze the training checkpoints of different-sized OPT models on next-token prediction, sequence-level generation and downstream tasks. |
| Outcome: | The results show that language models of different sizes learn more during training . small models halt at hallucinations, larger ones learn to assign lower probabilities . |
MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning (2021.naacl-main)
Copied to clipboard
Mengzhou Xia, Guoqing Zheng, Subhabrata Mukherjee, Milad Shokouhi, Graham Neubig, Ahmed Hassan Awadallah
| Challenge: | Recent work shows that multilingual representations are disjointed across languages, bringing additional challenges for transfer onto extremely low-resource languages. |
| Approach: | They propose a meta-learning based framework that learns to transform representations judiciously from auxiliary languages to a target one and brings their representation spaces closer for effective transfer. |
| Outcome: | The proposed framework learns to transform representations from auxiliary languages to a target language and brings their representation spaces closer for effective transfer. |