| Challenge: | RNNGs with syntactic supervision underperformed RNNs with some syntaktic supervision, whereas RNNS with mild supervision achieved the best performance comparable to the state-of-the-art GPT-2-XL. |
| Approach: | They propose a method where syntactic LMs are gradually ablated from full syntatic supervision to zero syntastic supervision by preserving NP, VP, PP, SBAR nonterminal symbols. |
| Outcome: | The proposed method underperforms the RNNGs with zero syntactic supervision, and the LMs with mild syntaktic supervision perform better than the state-of-the-art GPT-2-XL. |
Similar Papers
Decoding with Limited Teacher Supervision Requires Understanding When to Trust the Teacher (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated their tremendous capability to generate human-like text sentences that convey rich knowledge in various problem domains. |
| Approach: | They propose an algorithm to aggregate small-scale LLM and LLM predictions on initial tokens so that the generated tokens can more accurately condition the subsequent token generation by small-level LLM only. |
| Outcome: | The proposed method improves on the limited supervision scenario on a wide range of models and datasets. |
Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation (2021.eacl-main)
Copied to clipboard
| Challenge: | Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU). |
| Approach: | They empirically examine the usefulness of supervised parsing for semantic LU in LM-pretrained transformer networks. |
| Outcome: | The proposed model is based on LM-pretrained transformer networks with a biaffine parsing head and fine-tuned for LU tasks. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have not investigated the relationship between a token's frequency in the training corpus and syntactic properties models learn about it. |
| Approach: | They develop controlled experiments that probe models’ syntactic nominal number and verbal argument structure generalizations for tokens seen as few as two times during training. |
| Outcome: | The proposed models can make syntactic generalizations for tokens seen as few as two times during training and transfer them to transformed contexts. |
Lower Bounds on the Expressivity of Recurrent Neural Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies of the representational capacity of neural LMs have focused on their ability to recognize formal languages. |
| Approach: | They propose to connect recurrent neural networks (RNNs) as classifiers to finite-state automatas (FSAs) and a probabilistic FSA to characterize their representational capacity. |
| Outcome: | The proposed models can express arbitrary regular LMs with linearly bounded precision. |
On the Benefit of Syntactic Supervision for Cross-lingual Transfer in Semantic Role Labeling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in neural architectures and pre-trained representations have greatly improved the performance of fully-supervised semantic role labeling (SRL) but there are limitations in the availability of supervised training data. |
| Approach: | They propose to leverage syntactic dependencies to facilitate cross-lingual transfer by annotating predicate-argument structures in text. |
| Outcome: | The proposed model can be extended to other languages with limited training data. |
The Limitations of Limited Context for Constituency Parsing (2021.acl-long)
Copied to clipboard
| Challenge: | a language model that is syntax-aware can produce better samples, authors say . a recent study shows that neural approaches to syntax can perform unsupervised syntactic parsing . |
| Approach: | They propose to incorporate syntax into neural approaches in NLP to produce better samples . they find that the first time neural approaches were able to perform unsupervised syntactic parsing . |
| Outcome: | The proposed models can perform unsupervised syntactic parsing, but they are lagging behind . the proposed models are based on a sandbox of probabilistic context-free-grammars . |
RNN Simulations of Grammaticality Judgments on Long-distance Dependencies (C18-1)
Copied to clipboard
| Challenge: | LSTM networks can detect linguistic structures which are ungrammatical due to extraction violations, but are sensitive to linguistic processing factors. |
| Approach: | They propose to use LSTM networks to detect ungrammatical sentences by detecting extra arguments and subject-relative clause island violations. |
| Outcome: | The proposed model can correctly classify (un)grammatical sentences, in certain conditions, but is sensitive to linguistic processing factors and unable to induce a more abstract notion of grammaticality. |
AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research (2025.acl-long)
Copied to clipboard
Yilun Zhao, Weiyuan Chen, Zhijian Xu, Manasi Patwardhan, Chengye Wang, Yixin Liu, Lovekesh Vig, Arman Cohan
| Challenge: | a benchmark designed to evaluate the capabilities of LLMs in designing ablation studies for scientific research is available online. |
| Approach: | They propose to use a benchmark to evaluate LLMs' ability to design ablation studies . they investigate whether current automated evaluation methods are not reliable . |
| Outcome: | The benchmark compared leading LLMs with human experts on generating detailed ablation study designs . the results show that current evaluation methods are not reliable for the task . |
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies (N19-1)
Copied to clipboard
| Challenge: | State-of-the-art LSTM language models learn sequential contingencies with some success . LS models fail to learn other non-local grammatical dependencies, however . |
| Approach: | They compare LSTM language models with RNNGs to examine grammatical dependencies . they find that hierarchical supervision improves learning of non-local dependencies. |
| Outcome: | The proposed model outperforms the existing model on non-local dependencies and learns many of the Island Constraints on the filler-gap dependency. |