Papers by Shawn Tan
Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle (2021.naacl-main)
Copied to clipboard
| Challenge: | Failing to capture the structure of input language could lead to generalization problems and over-parametrization. |
| Approach: | They propose a new syntax-aware language model that explicitly models the structure with an incremental parser and maintains the conditional probability setting of a standard language model. |
| Outcome: | The proposed model can achieve strong results in language modeling, parsing, and syntactic generalization tests while using fewer parameters than other models. |
Recursive Top-Down Production for Sentence Generation with Latent Trees (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Various studies have shown that incorporating syntactic structures into recursive encoders can be beneficial for various natural language tasks. |
| Approach: | They propose a dynamic programming algorithm that marginalises over latent binary tree structures with N leaves to train a recursive neural function. |
| Outcome: | The proposed model outperforms previous models on the LENGTH split and English question formation tasks on the Multi30k dataset. |
Sparse Universal Transformer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that use VTs as their backbone model are based on UTs that share parameters across layers and have better compositional generalization. |
| Approach: | They propose to use Sparse Mixture of Experts to reduce UT's computation complexity while retaining its parameter efficiency and generalization ability. |
| Outcome: | The proposed model achieves strong generalization results on formal language tasks and impressive parameter and computation efficiency on standard natural language benchmarks. |
Unsupervised Dependency Graph Network (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work has identified properties of pretrained self-attention models that mirror those of dependency parse structures. |
| Approach: | They propose a model that encourages attention heads to model different dependency relations from raw corpora and a masked language modeling task. |
| Outcome: | The proposed model can induce dependency structures from raw corpora and the masked language modeling task without gold POS tags and any external information. |