Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for extracting complete (binary) parses from pre-trained language models are expensive and time-consuming. |
| Approach: | They propose a chart-based method and an effective top-K ensemble technique to extractbinary parses from PLMs. |
| Outcome: | The proposed method can induce non-trivial parses for sentences from nine languages in an integrated and language-agnostic manner, and is robust to cross-lingual transfer. |
Similar Papers
Revisiting the Practical Effectiveness of Constituency Parse Extraction from Pre-trained Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Constituency Parse Extraction from Pre-trained Language Models (CPE-PLM) is a new paradigm that attempts to induce constituency parse trees based on the internal knowledge of pre-tried language models. |
| Approach: | They propose to use constituency parse trees from pre-trained language models to induce constituency trees by introducing a set of heterogeneous PLMs combined using two advanced ensemble methods. |
| Outcome: | The proposed approach is more effective than typical supervised parsers in few-shot settings. |
Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing approaches to analyze syntactic knowledge of pre-trained language models have been limited. |
| Approach: | They propose an unsupervised method that extracts constituency trees from PLM attention heads. |
| Outcome: | The proposed method outperforms existing approaches if no development set is present. |
Multilingual Constituency Parsing with Self-Attention and Pre-Training (P19-1)
Copied to clipboard
| Challenge: | a range of pre-training conditions can be used for constituency parsing, but large model sizes make it expensive to train separate models for each language. |
| Approach: | They compare the benefits of no pre-training, fastText, ELMo, and BERT for English . they also find that pre- training is beneficial across all 11 languages tested . |
| Outcome: | The proposed model outperforms fastText, ELMo, and BERT for English . but large model sizes make it expensive to train separate models for each language . |
Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers (2023.acl-long)
Copied to clipboard
| Challenge: | a novel chart-based method for extracting parse trees from masked language models is proposed . a graph-based approach can be used to extract parser trees without training separate parsers . |
| Approach: | They propose a chart-based method for extracting parse trees from masked language models . they use a set of perturbations motivated by the linguistic concept of constituency tests to score each span . |
| Outcome: | The proposed method outperforms state-of-the-art methods on english with masked LMs and in multilingual settings. |
Mixed-Lingual Pre-training for Cross-lingual Summarization (2020.aacl-main)
Copied to clipboard
| Challenge: | Cross-lingual summarization (CLS) aims at producing a summary in the target language for an article in the source language. |
| Approach: | They propose a mixed-lingual pre-training scheme that leverages both cross-lingual tasks such as translation and monolingual tasks like masked language models. |
| Outcome: | The proposed model improves on the translation and masked language models with no task-specific components and saves memory. |
On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing decoder-based pre-trained language models demonstrate excellent multilingual capabilities, but it is unclear how they handle multilingualism. |
| Approach: | They propose to examine the neuron-level internal behavior of decoder-based PLMs by finding neurons that fire “uniquely for each language” within decoded PLM models. |
| Outcome: | The proposed models fire “uniquely for each language” and show that language-specific neurons are unique, with a slight overlap (5%) between languages. |
Rule Augmented Unsupervised Constituency Parsing (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that unsupervised parsing methods do not learn meaningful semantics (not even simple grammar) |
| Approach: | They propose an approach that utilizes very generic linguistic knowledge of the language present in the form of syntactic grammar rules and is independent of the base system. |
| Outcome: | The proposed model is independent of the base system and takes advantage of syntactic grammar rules. |
Frustratingly Simple but Surprisingly Strong: Using Language-Independent Features for Zero-shot Cross-lingual Semantic Parsing (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing training data is limited for languages other than English, so is the performance of the developed parsers. |
| Approach: | They propose to apply a pre-trained multilingual model to Italian, German and Dutch parsers where only a small number of manually annotated parses are available. |
| Outcome: | The proposed model improves on six parsers in English and Italian, German and Dutch, with the addition of universal dependency relations and universal POS tags as model-agnostic features. |
An Empirical Study of Building a Strong Baseline for Constituency Parsing (P18-2)
Copied to clipboard
| Challenge: | Sequence-to-sequence models have been used for natural language generation tasks such as machine translation and summarization. |
| Approach: | They propose to build a strong baseline based on general purpose sequence-to-sequence models for constituency parsing. |
| Outcome: | The proposed model outperforms existing models in natural language generation tasks without any explicit task-specific knowledge or architecture of constituent parsing. |
What’s Going On in Neural Constituency Parsers? An Analysis (N18-1)
Copied to clipboard
| Challenge: | a number of differences have emerged between classical and modern constituency parsing approaches . structural components like grammars and feature-rich lexicons are becoming less central . recurrent neural networks have gained traction as a powerful and general purpose tool for representation . |
| Approach: | They propose a model that implicitly learns to encode much of the same information as grammars and lexicons in the past. |
| Outcome: | The proposed model outperforms state-of-the-art models under similar conditions. |