Papers by Bingzhi Li
SLOG: A Structural Generalization Benchmark for Semantic Parsing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing compositional generalization benchmarks focus on lexical generalisation, the interpretation of novel lexicals in syntactic structures familiar from training. |
| Approach: | They propose a semantic parsing dataset that extends COGS with 17 structural generalization cases to evaluate how well models generalize to new complex linguistic expressions. |
| Outcome: | The proposed model generalization accuracy is far below the near-perfect accuracy of existing models on COGS, demonstrating the role of SLOG in foregrounding the large discrepancy between models’ lexical and structural generalization capacities. |
Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that unsupervised sentence representations of neural networks encode syntactic information by observing that neural language models are able to predict the agreement between a verb and its subject. |
| Approach: | They propose to take an alternative look at these results by studying whether neural networks are able to build an abstract sentence representation rather than capture surface statistical regularities. |
| Outcome: | The proposed model can achieve high accuracy on the long-range French object-verb agreement, indicating a possible flaw in the model's syntactic ability. |
Are Neural Networks Extracting Linguistic Properties or Memorizing Training Data? An Observation with a Multilingual Probe for Predicting Tense (2021.eacl-main)
Copied to clipboard
| Challenge: | a recent study has shown that neural networks can learn from linguistic representations without supervision . many studies have tried to identify which linguistic properties are encoded in the embeddings . |
| Approach: | They evaluate the ability of Bert embeddings to represent tense information . they use a multilingual linguistic probe to predict the morphology of a word . |
| Outcome: | The proposed model can predict tenses in French and Chinese, but the results drop sharply for Chinese. |
Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreement (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing studies have shown that transformers are able to predict subject-verb agreement, demonstrating their ability to uncover an abstract representation of the sentence in an unsupervised way. |
| Approach: | They propose to compare how transformers handle subject-verb and object-past participle agreements in French using probing and counterfactual analysis methods. |
| Outcome: | The proposed model handles subject-verb and object-past participle agreements in a way consistent with their modeling in theoretical linguistics. |
How Distributed are Distributed Representations? An Observation on the Locality of Syntactic Information in Verb Agreement Tasks (2022.acl-short)
Copied to clipboard
| Challenge: | Using probing, causal analysis and feature selection, we find that syntactic information is encoded locally in the transformers representations consistent with the French grammar. |
| Approach: | They address the question of the localization of syntactic information encoded in transformers representations by probing, causal analysis and feature selection methods. |
| Outcome: | The proposed representations are consistent with the object-past participle agreement in French and are consistent in both languages. |