Papers by Xuezhe Ma
Investigating the Benefits of Free-Form Rationales (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that crowdsourced rationales provide additional background knowledge to models . a qualitative study shows generated rationale is not as useful for humans as crowdsourced ones . |
| Approach: | They investigate whether crowdsourced rationales provide additional background knowledge to models . they find that ECQA rationale provides additional background information to understand a decision . |
| Outcome: | The results show that ECQA rationales provide additional background knowledge to understand a decision . compared to crowdsourced rationale, generated rationale is not as useful for humans . |
Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing fine-tuning algorithms for vision-language models are restricted by patient privacy concerns and can contain imperceptible noise. |
| Approach: | They propose a framework to mitigate adversarial noise and mitigate upstream noise during fine-tuning. |
| Outcome: | The proposed framework improves model robustness and transferability while decreasing noise levels negatively impact downstream performance. |
Look-back Decoding for Open-Ended Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to decode open-ended text have addressed degeneration problems in large-scale language models (LLMs) |
| Approach: | They propose an improved decoding algorithm that leverages the Kullback–Leibler divergence to track the distribution distance between current and historical decoding steps. |
| Outcome: | The proposed algorithm outperforms existing methods in document continuation and story generation. |
Handling Syntactic Divergence in Low-resource Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to neural machine translation (NMT) are dependent on limited parallel data, and can be difficult to use for many language pairs. |
| Approach: | They propose a method where target-language sentences are re-ordered to match the order of the source and used as an additional source of training-time supervision. |
| Outcome: | The proposed method improves on simulated low-resource Japanese-to-English and real low-demand Uyghur-to English scenarios. |
COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in pretrained language models have shown promising results on commonsense reasoning benchmark datasets. |
| Approach: | They propose a commonsense reasoning benchmark dataset with 4k sentence pairs . they propose 'gamified' model-in-the-loop setup to incentivize challenging samples . |
| Outcome: | The proposed benchmarks show that the proposed model achieves 71% standard accuracy and 51% pairwise accuracy, well below human performance. |
RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to personalized dialogue generation rely on dialogue data paired with user traits, profiles or persona description sentences. |
| Approach: | They propose a hierarchical transformer retriever trained on dialogue domain data to perform personalized retrieval and a context-aware prefix encoder that fuses the retrieved information to the decoder more effectively. |
| Outcome: | The proposed model generates more fluent and personalized responses under a suite of human and automatic metrics and is superior to state-of-the-art baselines on English Reddit conversations. |
Personalized Response Generation via Generative Split Memory Network (2021.naacl-main)
Copied to clipboard
| Challenge: | Despite the success of text generation and dialogue systems, how to endow a text generation system with personality traits remains under-investigated. |
| Approach: | They propose a model to generate personalized responses on reddit using user profiles and posting histories. |
| Outcome: | The proposed model improves over the state-of-the-art response generation models. |
Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation (P19-3)
Copied to clipboard
Zhiting Hu, Haoran Shi, Bowen Tan, Wentao Wang, Zichao Yang, Tiancheng Zhao, Junxian He, Lianhui Qin, Di Wang, Xuezhe Ma, Zhengzhong Liu, Xiaodan Liang, Wanrong Zhu, Devendra Sachan, Eric Xing
| Challenge: | Texar is an open-source text generation toolkit that supports a broad set of text generation tasks. |
| Approach: | They introduce Texar, an open-source text generation toolkit that supports text generation tasks. |
| Outcome: | Texar supports machine translation, summarization, dialog, content manipulation, and more. |
AESOP: Paraphrase Generation with Adaptive Syntactic Control (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for paraphrase generation use fixed syntactic structures for all input sentences. |
| Approach: | They propose to add syntactical control to a pretrained language model to generate fluent paraphrases using a retrieval-based selection module. |
| Outcome: | The proposed model achieves state-of-the-art on semantic preservation and syntactic conformation on two benchmark datasets with ground-truth syntaktic control from human-annotated exemplars. |
Evaluating Large Language Models on Controlled Generation Tasks (2023.emnlp-main)
Copied to clipboard
Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, Xuezhe Ma
| Challenge: | Recent studies have looked into the ability of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc. However, few studies investigate the controllability of large languages. |
| Approach: | They propose to compare large language models with state-of-the-start finetuned smaller models to find that large language model controls are comparable to smaller models. |
| Outcome: | The proposed model can meet hard constraints and perform better than state-of-the-art models. |
Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs. |
| Approach: | They propose a component-wise gradient norm clipping method to adjust convergence speed for different components to improve generalization performance, convergence speed, and training stability. |
| Outcome: | The proposed method achieves consistent improvements in terms of generalization performance, convergence speed, and training stability. |
DiSCoL: Toward Engaging Dialogue Systems through Conversational Line Guided Response Generation (2021.naacl-demos)
Copied to clipboard
| Challenge: | Recent advances in transformer-based language models and their applications to dialogue systems have succeeded to generate fluent and human-like responses. |
| Approach: | They propose to leverage conversational lines as controllable and informative content-planning elements to guide the generation model produce engaging and informative responses. |
| Outcome: | The proposed system generates engaging and informative responses using convlines as controllable and informative content-planning elements. |
An Empirical Investigation of Structured Output Modeling for Graph-based Neural Dependency Parsing (P19-1)
Copied to clipboard
| Challenge: | In the past few years, graph-based dependency parsers have led to impressive empirical successes on parsing accuracy. |
| Approach: | They propose to use a graph-based dependency parser to model global outputs. |
| Outcome: | The proposed model has been shown to perform better on sentence-level Complete Match metric compared with the previous model. |
Know Where You’re Going: Meta-Learning for Parameter-Efficient Fine-Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on parameter-efficient fine-tuning methods require additional measures after pre-training and before fine-uning. |
| Approach: | They propose to take parameter-efficient fine-tuning into consideration after pre-training and before fine-uning and use meta-learning to prime a model specifically for parameter-efficiency. |
| Outcome: | The proposed method improves on a pre-trained model with certain modifications and achieves 4.96 points on cross-lingual NER fine-tuning. |
Density Matching for Bilingual Word Embedding (N19-1)
Copied to clipboard
| Challenge: | Recent approaches to cross-lingual word embeddings have been based on linear transformations between the embeddable vectors in the two languages. |
| Approach: | They propose a method that expresses two monolingual embedding spaces as probability densities and matches them using a Gaussian mixture model. |
| Outcome: | The proposed method can achieve competitive or superior performance on bilingual lexicon induction and cross-lingual word similarity data. |
On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing (N19-1)
Copied to clipboard
| Challenge: | Existing studies on crosslingual transfer have focused on word-level information sharing, but words are not independent in sentences; their combinations form larger linguistic units, known as context. |
| Approach: | They propose to use orderagnostic models to transfer word order to distant languages . they train dependency parsers on an English corpus and evaluate their transfer performance on 30 other languages. |
| Outcome: | The proposed model performs better on languages with different word orders than on other languages. |
Prompt Consistency for Zero-Shot Task Generalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has shown that pre-trained language models can perform zero-shot generalization to new tasks without annotated examples. |
| Approach: | They propose to regularize prompt consistency to encourage consistent predictions over a diverse set of prompts. |
| Outcome: | The proposed approach outperforms the state-of-the-art zero-shot learner, T0, on 9 out of 11 datasets across 4 NLP tasks by 10.6 absolute points in terms of accuracy. |
A Two-Step Approach for Implicit Event Argument Detection (2020.acl-main)
Copied to clipboard
| Challenge: | et al., 2015) only consider local arguments in the same sentence of the event trigger. |
| Approach: | They propose to decompose the implicit event argument detection task into two sub-problems . they propose to use argument head-word detection and head-to-span expansion to reduce the number of candidates. |
| Outcome: | The proposed model achieves better performance than a strong sequence labeling baseline. |
LLM The Genius Paradox: A Linguistic and Math Expert’s Struggle with Simple Word-based Counting Problems (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing conjectures about the reason for deficiency of LLMs in simple word-based counting problems are invalid. |
| Approach: | They propose to evaluate model transferability from specialized LLMs to simple counting tasks by comparing their results to popular conjectures . |
| Outcome: | The proposed model evaluations show that engaging reasoning is the most robust and efficient way to help LLMs better perceive tasks with more accurate responses. |
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow (D19-1)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models are autoregressive, meaning they factor the joint probability of the output sequence into the product of probabilities over the next to-ken. |
| Approach: | They propose a non-autoregressive sequence generation model using latent variables . they use generative flow to model complex distributions using neural networks . |
| Outcome: | The proposed model performs comparable to state-of-the-art models and has constant decoding time w.r.t the sequence length. |
Challenges in Context-Aware Neural Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | despite well-reasoned intuitions, most context-aware neural machine translation models show only modest improvements over sentence-level systems. |
| Approach: | They propose a more realistic setting for document-level translation called paragraph-to-paragraph (PARA2PARA) they collect a dataset of Chinese-English novels to promote future research . |
| Outcome: | The proposed model improves translation quality across document-level metrics and discourse phenomena. |
Stack-Pointer Networks for Dependency Parsing (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to dependency parsing are local and greedy transitionbased . StackPtr parsers use the information of whole sentences and previously derived subtree structures . |
| Approach: | They propose a stack-pointer network-based dependency parser that reads whole sentence and builds dependency tree top-down in a depth-first fashion. |
| Outcome: | The proposed model reads and encodes whole sentence, then builds dependency tree top-down (from root-to-leaf) in a depth-first fashion. |
Choosing Transfer Languages for Cross-Lingual Learning (P19-1)
Copied to clipboard
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, Graham Neubig
| Challenge: | Cross-lingual transfer is a useful tool for improving performance of natural language processing (NLP) on low-resource languages. |
| Approach: | They propose to use cross-lingual transfer to improve accuracy of low-resource languages . they build models that consider features to perform prediction on such languages based on ranking problem . |
| Outcome: | The proposed model predicts good transfer languages much better than baselines considering single features in isolation. |