Papers by Chunting Zhou
Look-back Decoding for Open-Ended Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to decode open-ended text have addressed degeneration problems in large-scale language models (LLMs) |
| Approach: | They propose an improved decoding algorithm that leverages the Kullback–Leibler divergence to track the distribution distance between current and historical decoding steps. |
| Outcome: | The proposed algorithm outperforms existing methods in document continuation and story generation. |
Handling Syntactic Divergence in Low-resource Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to neural machine translation (NMT) are dependent on limited parallel data, and can be difficult to use for many language pairs. |
| Approach: | They propose a method where target-language sentences are re-ordered to match the order of the source and used as an additional source of training-time supervision. |
| Outcome: | The proposed method improves on simulated low-resource Japanese-to-English and real low-demand Uyghur-to English scenarios. |
Adapting Word Embeddings to New Languages with Morphological and Phonological Subword Representations (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to generalization to resource-rich languages are difficult . a recent study shows that word representations can be useful in low resource languages . |
| Approach: | They propose two approaches for improving generalization to low-resource languages by adapting continuous word representations using linguistically motivated subword units. |
| Outcome: | The proposed method improves generalization to low resource languages . it requires neither parallel corpora nor bilingual dictionaries and requires no parallel training . |
MART: Improving LLM Safety with Multi-round Automatic Red-Teaming (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing red-teaming methods for large language models often discover safety risks without addressing them. |
| Approach: | They propose a multi-round automatic red-teaming method that incorporates both adversarial prompt writing and safe response generation. |
| Outcome: | The proposed method significantly increases red-teaming scalability and the safety of the target LLM. |
StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing (P18-1)
Copied to clipboard
| Challenge: | Semantic parsing is the task of transducing natural language (NL) utterances into formal meaning representations (MRs), commonly represented as tree structures. |
| Approach: | They propose a variational auto-encoding model for semi-supervised semantic parsing which learns from limited amounts of parallel data and readily-available unlabeled NL utterances. |
| Outcome: | Experiments on ATIS domain and Python show that with extra unlabeled data, StructVAE outperforms strong supervised models. |
Multi-Dimensional Evaluation of Text Summarization with In-Context Learning (2023.findings-acl)
Copied to clipboard
Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, Chunting Zhou
| Challenge: | In-context learning-based evaluators are competitive with learned evaluation frameworks for text summarization tasks. |
| Approach: | They propose to use large language models as multi-dimensional evaluators using in-context learning to evaluate text summarization tasks. |
| Outcome: | The proposed frameworks are competitive with existing frameworks on relevance and factual consistency, the authors show . |
Instruction-tuned Language Models are Better Knowledge Learners (2024.acl-long)
Copied to clipboard
Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Lin, Wen-tau Yih, Srini Iyer
| Challenge: | Large language models store factual knowledge in parameters, but it can become outdated as the work evolves . pre-instruction-tuning improves ability of LLMs to absorb knowledge from new documents . |
| Approach: | They propose a method that instruction-tunes on questions prior to training on documents . they propose to use QA pairs to update factual knowledge of large language models . |
| Outcome: | The proposed method outperforms instruction-tuning on documents by 17.8%. |
Density Matching for Bilingual Word Embedding (N19-1)
Copied to clipboard
| Challenge: | Recent approaches to cross-lingual word embeddings have been based on linear transformations between the embeddable vectors in the two languages. |
| Approach: | They propose a method that expresses two monolingual embedding spaces as probability densities and matches them using a Gaussian mixture model. |
| Outcome: | The proposed method can achieve competitive or superior performance on bilingual lexicon induction and cross-lingual word similarity data. |
Byte Latent Transformer: Patches Scale Better Than Tokens (2025.acl-long)
Copied to clipboard
Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, Srini Iyer
| Challenge: | Existing large language models (LLMs) are trained on bytes, except for tokenization, which groups bytes into a static set of tokens. |
| Approach: | They propose a new byte-level LLM architecture that encodes bytes into dynamically sized patches, which serve as the primary units of computation. |
| Outcome: | The proposed architecture matches tokenization-based models with improvements in inference efficiency and robustness. |
In-context Examples Selection for Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale generative models can perform a wide range of NLP tasks using in-context learning. |
| Approach: | They aim to understand the properties of good in-context examples for machine translation in both in-domain and out-of-domain settings. |
| Outcome: | The proposed model outperforms a strong kNN-MT baseline in 2 out of 4 out-of-domain datasets. |
Prompt Consistency for Zero-Shot Task Generalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has shown that pre-trained language models can perform zero-shot generalization to new tasks without annotated examples. |
| Approach: | They propose to regularize prompt consistency to encourage consistent predictions over a diverse set of prompts. |
| Outcome: | The proposed approach outperforms the state-of-the-art zero-shot learner, T0, on 9 out of 11 datasets across 4 NLP tasks by 10.6 absolute points in terms of accuracy. |
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow (D19-1)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models are autoregressive, meaning they factor the joint probability of the output sequence into the product of probabilities over the next to-ken. |
| Approach: | They propose a non-autoregressive sequence generation model using latent variables . they use generative flow to model complex distributions using neural networks . |
| Outcome: | The proposed model performs comparable to state-of-the-art models and has constant decoding time w.r.t the sequence length. |
Detecting Hallucinated Content in Conditional Neural Sequence Generation (2021.findings-acl)
Copied to clipboard
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, Marjan Ghazvininejad
| Challenge: | Neural sequence models can generate fluent sentences, but they can also hallucinate additional content not supported by the input. |
| Approach: | They propose a task to predict whether each token in the output sequence is hallucinated and collect manually annotated evaluation sets for this task. |
| Outcome: | The proposed method outperforms baseline methods on machine translation and abstractive summarization datasets and achieves significant improvements in both supervised and unsupervised settings. |
Distributionally Robust Multilingual Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual neural machine translation (MNMT) learns to translate multiple language pairs with a single model, but the data imbalance hinders it from performing uniformly across language pairs. |
| Approach: | They propose a distributionally robust optimization objective which minimizes the worst-case expected loss over the set of language pairs. |
| Outcome: | The proposed learning objective outperforms baseline methods on three sets of languages and shows that it is cost-effective and efficient. |
Training Trajectories of Language Models Across Scales (2023.acl-long)
Copied to clipboard
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. |
| Approach: | They analyze the training checkpoints of different-sized OPT models on next-token prediction, sequence-level generation and downstream tasks. |
| Outcome: | The results show that language models of different sizes learn more during training . small models halt at hallucinations, larger ones learn to assign lower probabilities . |