Papers by Yikang Shen

Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle (2021.naacl-main)

Copied to clipboard

Yikang Shen, Shawn Tan, Alessandro Sordoni, Siva Reddy, Aaron Courville

Challenge:	Failing to capture the structure of input language could lead to generalization problems and over-parametrization.
Approach:	They propose a new syntax-aware language model that explicitly models the structure with an incremental parser and maintains the conditional probability setting of a standard language model.
Outcome:	The proposed model can achieve strong results in language modeling, parsing, and syntactic generalization tests while using fewer parameters than other models.

Recursive Top-Down Production for Sentence Generation with Latent Trees (2020.findings-emnlp)

Copied to clipboard

Shawn Tan, Yikang Shen, Alessandro Sordoni, Aaron Courville, Timothy J. O’Donnell

Text Generation

Challenge:	Various studies have shown that incorporating syntactic structures into recursive encoders can be beneficial for various natural language tasks.
Approach:	They propose a dynamic programming algorithm that marginalises over latent binary tree structures with N leaves to train a recursive neural function.
Outcome:	The proposed model outperforms previous models on the LENGTH split and English question formation tasks on the Multi30k dataset.

StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling (2021.acl-long)

Copied to clipboard

Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, Aaron Courville

Language Models

Challenge:	Existing models that induce grammar structures from data focus on constituency or dependency structures alone.
Approach:	They propose a model that can induce dependency and constituency structure at the same time.
Outcome:	The proposed model can induce both constituency and dependency structures at the same time.

A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese (2026.tacl-1)

Copied to clipboard

Yikang Liu, Yeting Shen, Hongao Zhu, Lilong Xu, Zhiheng Qian, Siyuan Song, Kejia Zhang, Jialong Tang, Pei Zhang, Baosong Yang, Rui Wang, Hai Hu

Semantic Text Processing

Challenge:	Using sub-linear length normalized log-probabilities (SLLN-LP), we find unequal lengths of sentences in minimal pairs difficult for LMs even up to 32B parameters.
Approach:	They propose to use ZhoBLiMP as a linguistic minimal pair benchmark for Chinese language models to mitigate biases.
Outcome:	The proposed metric mitigates biases in Chinese language models with over 100 paradigms . Anaphor, Quantifiers, and Ellipsis are difficult for LMs even up to 32B parameters .

Sparse Universal Transformer (2023.emnlp-main)

Copied to clipboard

Shawn Tan, Yikang Shen, Zhenfang Chen, Aaron Courville, Chuang Gan

Semantic Text Processing

Challenge:	Existing models that use VTs as their backbone model are based on UTs that share parameters across layers and have better compositional generalization.
Approach:	They propose to use Sparse Mixture of Experts to reduce UT's computation complexity while retaining its parameter efficiency and generalization ability.
Outcome:	The proposed model achieves strong generalization results on formal language tasks and impressive parameter and computation efficiency on standard natural language benchmarks.

Straight to the Tree: Constituency Parsing with Neural Syntactic Distance (P18-1)

Copied to clipboard

Yikang Shen, Zhouhan Lin, Athul Paul Jacob, Alessandro Sordoni, Aaron Courville, Yoshua Bengio

Syntactic Text Processing

Challenge:	Compared to traditional shift-reduce parsing schemes, our approach is free from the potentially disastrous compounding error.
Approach:	They propose a model that predicts a scalar for each split position in a sentence and then determines the topology of grammar tree based on syntactic distances.
Outcome:	The proposed model achieves the state-of-the-art single model F1 score of 92.1 on PTB and 86.4 on CTB dataset, surpassing the previous single model results by a large margin.

Unsupervised Dependency Graph Network (2022.acl-long)

Copied to clipboard

Yikang Shen, Shawn Tan, Alessandro Sordoni, Peng Li, Jie Zhou, Aaron Courville

Language Models

Challenge:	Recent work has identified properties of pretrained self-attention models that mirror those of dependency parse structures.
Approach:	They propose a model that encourages attention heads to model different dependency relations from raw corpora and a masked language modeling task.
Outcome:	The proposed model can induce dependency structures from raw corpora and the masked language modeling task without gold POS tags and any external information.

Phrase-aware Unsupervised Constituency Parsing (2022.acl-long)

Copied to clipboard

Xiaotao Gu, Yikang Shen, Jiaming Shen, Jingbo Shang, Jiawei Han

Low-Resource NLP

Challenge:	Recent studies have achieved inspiring success in unsupervised grammar induction using masked language modeling (MLM) as the proxy task.
Approach:	They propose to regularize the parser with phrases extracted by an unsupervised phrase tagger to help the LM model quickly manage low-level structures.
Outcome:	The proposed method improves the identification of high-level structures using phrase-guided masking.

Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach (2020.acl-main)

Copied to clipboard

Wenyu Du, Zhouhan Lin, Yikang Shen, Timothy J. O’Donnell, Yoshua Bengio, Yue Zhang

Syntactic Text Processing

Challenge:	incorporating syntactic structure into language models has been a challenge since the 1990s.
Approach:	They propose to use syntactic information to integrate syntastic structure into neural language models by providing ground truth parse trees as additional training signals.
Outcome:	The proposed model achieves lower perplexity and better quality when ground truth parse trees are provided as training signals.

Aligning Large Multimodal Models with Factually Augmented RLHF (2024.findings-acl)

Copied to clipboard

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell

Multimodality

Challenge:	Large Multimodal Models (LMMs) are built across modalities and the misalignment between two modality can result in "hallucination" . developing LMMs faces challenges such as a lack of data and a limited number of data sets.
Approach:	They propose a new algorithm that augments the reward model with additional factual information such as image captions and ground-truth multi-choice options.
Outcome:	The proposed approach improves on the LLaVA-Bench dataset with the 96% performance level of the text-only GPT-4 and an improvement of 60% on MMHAL-BENCH over other baselines.

Mixture of Attention Heads: Selecting Attention Heads Per Token (2022.emnlp-main)

Copied to clipboard

Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, Zhang Xiong

Responsible & Trustworthy NLP

Challenge:	Mixture-of-Experts (MoE) networks have been proposed as an efficient way to scale up model capacity and implement conditional computing.
Approach:	They propose a new architecture that combines multi-head attention with the MoE mechanism and a sparsely gated architecture that allows for faster computations.
Outcome:	The proposed architecture can scale up the number of attention heads and the number parameters while preserving computational efficiency.

BanditSum: Extractive Summarization as a Contextual Bandit (D18-1)

Copied to clipboard

Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, Jackie Chi Kit Cheung

Information Extraction & Text Mining

Challenge:	Existing methods for extractive summarization are heuristically generated and require a set of binary labels to be selected.
Approach:	They propose a method for training neural networks to perform single-document extractive summarization without heuristically-generated extractive labels.
Outcome:	The proposed method achieves better ROUGE scores than the state-of-the-art methods and significantly fewer update steps than competing approaches.