Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers (2023.acl-long)
Copied to clipboard
| Challenge: | a novel chart-based method for extracting parse trees from masked language models is proposed . a graph-based approach can be used to extract parser trees without training separate parsers . |
| Approach: | They propose a chart-based method for extracting parse trees from masked language models . they use a set of perturbations motivated by the linguistic concept of constituency tests to score each span . |
| Outcome: | The proposed method outperforms state-of-the-art methods on english with masked LMs and in multilingual settings. |
Similar Papers
Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for extracting complete (binary) parses from pre-trained language models are expensive and time-consuming. |
| Approach: | They propose a chart-based method and an effective top-K ensemble technique to extractbinary parses from PLMs. |
| Outcome: | The proposed method can induce non-trivial parses for sentences from nine languages in an integrated and language-agnostic manner, and is robust to cross-lingual transfer. |
Revisiting the Practical Effectiveness of Constituency Parse Extraction from Pre-trained Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Constituency Parse Extraction from Pre-trained Language Models (CPE-PLM) is a new paradigm that attempts to induce constituency parse trees based on the internal knowledge of pre-tried language models. |
| Approach: | They propose to use constituency parse trees from pre-trained language models to induce constituency trees by introducing a set of heterogeneous PLMs combined using two advanced ensemble methods. |
| Outcome: | The proposed approach is more effective than typical supervised parsers in few-shot settings. |
Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing approaches to analyze syntactic knowledge of pre-trained language models have been limited. |
| Approach: | They propose an unsupervised method that extracts constituency trees from PLM attention heads. |
| Outcome: | The proposed method outperforms existing approaches if no development set is present. |
Unsupervised Parsing via Constituency Tests (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for unsupervised parsing rely on constituency tests . linguists can judge a sentence's grammatical validity by modifying it via some transformation . |
| Approach: | They propose a method for unsupervised parsing based on a constituency test . they specify a set of transformations and use an unsupervised neural acceptability model to make grammaticality decisions. |
| Outcome: | The proposed method achieves 62.8 F1 on the Penn Treebank test set, an improvement of 7.6 points over the previous best results. |
Masked Language Model Scoring (2020.acl-main)
Copied to clipboard
| Challenge: | Pretrained masked language models require finetuning for most tasks. |
| Approach: | They evaluate pretrained masked language models out of the box via their pseudo-log-likelihood scores (PLLs) they attribute this success to PLL’s unsupervised expression of linguistic acceptability without a left-to-right bias, greatly improving on scores from GPT-2 . |
| Outcome: | The proposed model outperforms autoregressive language models in a variety of tasks. |
Parsing Headed Constituencies (2024.lrec-main)
Copied to clipboard
| Challenge: | Using constituency and dependency trees, syntactic representations are preferred for tasks such as nominal phrase extraction and identification of terminology. |
| Approach: | They propose a parsing technique that generates headed constituency trees which combine information typically contained in constituency and dependency trees. |
| Outcome: | The proposed method generates headed constituency trees with discontinuities and can generate constituency tree with discontinuity. |
BERT-Proof Syntactic Structures: Investigating Errors in Discontinuous Constituency Parsing (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent results show that pretrained language models can be used for many tasks with high accuracy and high performance. |
| Approach: | They propose two methods for automatically analysing discontinuous parsers' errors. |
| Outcome: | The proposed methods characterize errors of a state-of-the-art transition-based discontinuous parser and provide an overview of the contribution of BERT to this task. |
On Parsing as Tagging (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to reduce constituency parsing to tagging are based on linearization, learning, and decoding . linearization of the derivation tree is the most critical factor in achieving accurate parsers as taggers . |
| Approach: | They propose a pipeline with three steps for reducing constituency parsing to tagging . they find that linearization and learning are critical factors for accurate parsers . |
| Outcome: | The proposed pipelines are linearized, learning, and decoded, and have three steps to achieve accurate parsing as taggers. |
Span-based discontinuous constituency parsing: a family of exact chart-based algorithms with time complexities from O(nˆ6) down to O(nˆ3) (2020.emnlp-main)
Copied to clipboard
| Challenge: | a novel chart-based parser for discontinuous constituency trees is proposed for span-based span parsing . it can process discontinuous constituent trees of block degree two, including ill-nested structures . |
| Approach: | They propose a chart-based algorithm for span-based parsing of discontinuous constituency trees . they build variants with smaller search spaces and time complexities ranging from O(n6) down to O(N3) . |
| Outcome: | The proposed algorithm can process 98% of constituents in linguistic treebanks while having the same complexity as continuous constituency parsers. |
Phrase-aware Unsupervised Constituency Parsing (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have achieved inspiring success in unsupervised grammar induction using masked language modeling (MLM) as the proxy task. |
| Approach: | They propose to regularize the parser with phrases extracted by an unsupervised phrase tagger to help the LM model quickly manage low-level structures. |
| Outcome: | The proposed method improves the identification of high-level structures using phrase-guided masking. |