Papers by Shijie Wu
Exact Hard Monotonic Attention for Character-Level Transduction (P19-1)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models with soft attention outperform monotonic models . current dominant method is the neural sequenceto-Sequency model with soft focus . |
| Approach: | They develop a hard attention sequence-to-sequence model that enforces strict monotonicity and learns alignment jointly. |
| Outcome: | The proposed model achieves state-of-the-art on grapheme-to-phoneme conversion and morphological inflection generation. |
PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning (2026.acl-long)
Copied to clipboard
Jingcheng Hu, Yinmin Zhang, Shijie Shang, Xiaobo Yang, Yue Peng, Zhewei Huang, Hebin Zhou, Xin Wu, Jie Cheng, Fanqi Wan, Xiangwen Kong, Chengyuan Yao, Kaiwen Yan, Ailin Huang, Hongyu Zhou, Qi Han, Zheng Ge, Xiangyu Zhang, Heung-Yeung Shum
| Challenge: | Parallel Coordinated Reasoning (PaCoRe) overcomes a central limitation of contemporary language models: their inability to scale test-time compute (TTC) far beyond sequential reasoning under a fixed context window. |
| Approach: | They propose a training-and-inference framework to overcome a central limitation of language models: their inability to scale test-time compute (TTC) under a fixed context window. |
| Outcome: | The proposed model scales to multi-million-token effective TTC without exceeding context limits. |
Towards a Unified Multi-Domain Multilingual Named Entity Recognition Model (2023.eacl-main)
Copied to clipboard
Mayank Kulkarni, Daniel Preotiuc-Pietro, Karthik Radhakrishnan, Genta Indra Winata, Shijie Wu, Lingjue Xie, Shaohua Yang
| Challenge: | Named Entity Recognition is a key task whose performance is sensitive to genre and language. |
| Approach: | They propose a setup for Named Entity Recognition which includes multi-domain and multilingual training and evaluation across 13 domains and 4 languages. |
| Outcome: | The proposed model improves on 13 domains and 4 languages across 13 domain and 4 language domains. |
Do Explicit Alignments Robustly Improve Multilingual Encoders? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Explicit alignment objectives based on bitexts like Europarl and MultiUN have been shown to improve cross-lingual representations. |
| Approach: | They propose a new contrastive alignment objective that can better utilize bitexts . they propose to use a random sample of 1 million pair subset of OPUS data . |
| Outcome: | The proposed objective outperforms existing alignment objectives on a random 1 million pair subset of the OPUS dataset. |
Bernice: A Multilingual Pre-trained Encoder for Twitter (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing language models for Twitter are monolingual, adapted from other domains, or trained on limited amount of in-domain data. |
| Approach: | They propose a multilingual RoBERTa language model that is trained from scratch on 2.5 billion tweets with a custom tweet-focused tokenizer. |
| Outcome: | The proposed model outperforms or matches models trained on monolingual and multilingual tweets on a variety of benchmarks and is more efficient compute- and data-wise to train completely on in-domain data with a specialized domain-specific tokenizer. |
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (D19-1)
Copied to clipboard
| Challenge: | Pretrained contextual representation models have pushed forward the state-of-the-art on many NLP tasks. |
| Approach: | They propose to use a model that is pretrained on 104 languages for cross-lingual transfer. |
| Outcome: | The proposed model performs well on 5 NLP tasks covering 39 languages from various language families. |
Emerging Cross-lingual Structure in Pretrained Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has shown that multilingual pretraining works, but is unable to measure these effects. |
| Approach: | They propose to use multilingual masked language modeling to train a model on concatenated text from multiple languages to find universal latent symmetries in embedding spaces. |
| Outcome: | The proposed models can be trained on concatenated text from multiple languages without shared vocabulary or domain similarity. |
A Simple Joint Model for Improved Contextual Neural Lemmatization (N19-1)
Copied to clipboard
| Challenge: | False positive: a core NLP task of lemmatization seeks to map multiple forms of English verbs to a canonical one, known as the lemma. |
| Approach: | They propose a joint neural model for lemmatization and morphological tagging that achieves state-of-the-art results on 20 languages from the Universal Dependencies corpora. |
| Outcome: | The proposed model achieves state-of-the-art results on 20 languages from the Universal Dependencies corpora. |
Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning (2023.findings-acl)
Copied to clipboard
Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, Daniel Preotiuc-Pietro
| Challenge: | Existing methods to handle catastrophic forgetting fail to retain knowledge learnt in the past when sudden shifts occur in training data distributions. |
| Approach: | They propose a learning rate scheduling method that preserves new information without strongly overwriting past knowledge. |
| Outcome: | The proposed method preserves new information without overwriting past knowledge in a multilingual continuous learning framework. |
Academics Can Contribute to Domain-Specialized Language Models (2024.emnlp-main)
Copied to clipboard
Mark Dredze, Genta Winata, Prabhanjan Kambadur, Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, David Rosenberg, Sebastian Gehrmann
| Challenge: | Commercially available models dominate academic leaderboards, focusing on creating and adapting general-purpose models . however, general- purpose models often underperform in specialized domains, and domain-specific models yield superior results. |
| Approach: | They advocate for a renewed focus on developing and evaluating domain- and task-specific models . they advocate for an adapted or adapted model that can be used to improve academic leaderboard standings . |
| Outcome: | The proposed model can do well on professional and linguistic examinations, college-level knowledge questions, and collections of reasoning tasks. |
Cross-lingual Few-Shot Learning on Unseen Languages (2022.aacl-main)
Copied to clipboard
| Challenge: | Large pre-trained language models have demonstrated the ability to obtain good performance on downstream tasks with limited examples in resource-rich languages. |
| Approach: | They propose to use a downstream sentiment analysis task to analyze the effectiveness of several few-shot learning strategies across 12 languages, including 8 unseen languages, to compare results. |
| Outcome: | The proposed model, XLM-R, gives the best performance on a task with few examples in resource-rich languages. |
Which *BERT? A Survey Organizing Contextualized Encoders (2020.emnlp-main)
Copied to clipboard
| Challenge: | a survey on language representation learning aims to highlight common themes . we focus on the areas of progress, compared to other fields, and discuss how each area is evaluated. |
| Approach: | They present a survey on language representation learning to highlight common themes . they compare contributions in contextualized text encoders to ideas from other fields . |
| Outcome: | The proposed survey aims to highlight common themes in the field of language representation learning. |
The Paradigm Discovery Problem (2020.acl-main)
Copied to clipboard
| Challenge: | a paradigm discovery problem is a task of learning an inflectional morphological system from unannotated sentences. |
| Approach: | They formalize the paradigm discovery problem and develop evaluation metrics for judging systems . they use word embeddings and string similarity to cluster forms by cell and by paradigm . |
| Outcome: | The proposed system suggests clustering by cell across different inflection classes is the most pressing challenge for future work. |
MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies (2023.acl-long)
Copied to clipboard
| Challenge: | Autoregressive language models are trained by minimizing the cross-entropy of the model distribution Q relative to the data distribution P. However, these systems still struggle in many openended generation settings, where they are asked to produce a long text following a short prompt. |
| Approach: | They propose to combine forward and reverse cross-entropy to train autoregressive language models by minimizing the cross-Entropy of the model distribution Q relative to the data distribution P. |
| Outcome: | The proposed model overgeneralizes and produces non-human-like text without complex decoding strategies. |
Applying the Transformer to Character-level Transduction (2021.eacl-main)
Copied to clipboard
| Challenge: | morphological inflection generation and historical text normalization tasks are character-level tasks that outperform recurrent models. |
| Approach: | They propose a technique to handle feature-guided character-level transduction that further improves performance. |
| Outcome: | The transformer outperforms recurrent models on morphological inflection and historical text normalization tasks. |
Morphological Irregularity Correlates with Frequency (P19-1)
Copied to clipboard
| Challenge: | a new measure of irregularity is proposed for inflectional morphology of languages . the measure is based on the predictability of forms in a language . |
| Approach: | They propose a measure of irregularity based on the predictability of forms in a language. |
| Outcome: | The proposed measure is the first of its breadth and confirms longstanding proposals from the linguistics literature. |
Hard Non-Monotonic Attention for Character-Level Transduction (D18-1)
Copied to clipboard
| Challenge: | Character-level string-to-string transductions are an important component of NLP tasks . hard non-monotonic attention models have been used for sequence modeling tasks involving characters . |
| Approach: | They propose an exact algorithm for marginalizing over the exponential number of non-monotonic alignments between two strings. |
| Outcome: | The proposed algorithm outperforms soft attention and improves performance over stochastic approximation. |
AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts (2026.acl-long)
Copied to clipboard
Keyu Li, Junhao Shi, Yang Xiao, Mohan Jiang, Jie Sun, Yunze Wu, Dayuan Fu, Shijie Xia, Xiaojie Cai, Tianze Xu, Weiye Si, Wenjie Li, Dequan Wang, Pengfei Liu
| Challenge: | Existing benchmarks focus on single agentic capability, failing to capture long-horizon real-world scenarios. |
| Approach: | They propose a benchmark that evaluates 6 agentic capabilities across 32 real-world scenarios. |
| Outcome: | Experiments show that closed-source models outperform open-source model (48.4% vs 32.1%) integrating models with advanced scaffolds to form autonomous agents is a paradigm shift. |
Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction (2021.emnlp-main)
Copied to clipboard
Mahsa Yarmohammadi, Shijie Wu, Marc Marone, Haoran Xu, Seth Ebner, Guanghui Qin, Yunmo Chen, Jialiang Guo, Craig Harman, Kenton Murray, Aaron Steven White, Mark Dredze, Benjamin Van Durme
| Challenge: | Zero-shot cross-lingual information extraction (IE) is a technique for training data in a source language but not in . |
| Approach: | They explore techniques including data projection and self-training to improve zero-shot cross-lingual information extraction (IE) IE is a construction of an IE model for some target language given existing annotations exclusively in English. |
| Outcome: | The proposed techniques show that they perform better than any single strategy. |