| Challenge: | Recent work on structure-aware models have shown promising results on language modeling, but how to incorporate structure knowledge on corpus without syntactic annotations remains an open problem. |
| Approach: | They propose a neural variational language model which enables the sharing of grammar knowledge among different corpora. |
| Outcome: | The proposed model converges significantly faster to lower perplexity on two popular benchmark datasets. |
Similar Papers
Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach (2020.acl-main)
Copied to clipboard
| Challenge: | incorporating syntactic structure into language models has been a challenge since the 1990s. |
| Approach: | They propose to use syntactic information to integrate syntastic structure into neural language models by providing ground truth parse trees as additional training signals. |
| Outcome: | The proposed model achieves lower perplexity and better quality when ground truth parse trees are provided as training signals. |
Grammar Induction with Neural Language Models: An Unusual Replication (D18-1)
Copied to clipboard
| Challenge: | Recent work on latent tree learning attempts to develop models with parse-valued latent variables and train them on non-parsing tasks. |
| Approach: | They propose a model with parse-valued latent variables and a strong latent tree learning result on constituency parsing. |
| Outcome: | The proposed model outperforms all baselines and performs competitively with symbolic grammar induction systems. |
A Regularization-based Framework for Bilingual Grammar Induction (D19-1)
Copied to clipboard
| Challenge: | Existing multilingual grammar induction methods require external resources such as parallel corpora, word alignments or linguistic phylogenetic trees. |
| Approach: | They propose a framework in which the learning process of the grammar model of one language is influenced by knowledge from the model of another language. |
| Outcome: | The proposed method outperforms baselines on transfer grammar induction and bilingual grammar inducing on multiple languages. |
Improving Neural Language Models by Segmenting, Attending, and Predicting the Future (P19-1)
Copied to clipboard
| Challenge: | Common language models typically predict the next word given a past context. |
| Approach: | They propose a method that aligns the given context and the following phrase . they define syntactic heights and phrase segmentation rules to enable it to learn . |
| Outcome: | The proposed model outperforms strong baseline models on Wikitext-103 dataset. |
Incorporating Contextual and Syntactic Structures Improves Semantic Similarity Modeling (D19-1)
Copied to clipboard
| Challenge: | Semantic similarity modeling is central to many NLP problems such as question answering. |
| Approach: | They propose a pairwise word interaction model with syntactic structure priors to explore their effectiveness. |
| Outcome: | Extensive evaluations on eight benchmark datasets show that incorporating structural information improves over strong baselines. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
CoELM: Construction-Enhanced Language Modeling (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that integrating constructional information can improve the performance of pre-trained language models. |
| Approach: | They propose a construction-Enhanced language model that embeds constructional semantics into language models for natural language generation. |
| Outcome: | The proposed model outperforms existing models on various benchmarks. |
Pivot Based Language Modeling for Improved Neural Domain Adaptation (N18-1)
Copied to clipboard
| Challenge: | Existing work on domain adaptation does not exploit the structure of the input text . PBLM can naturally feed structure aware text classifiers such as LSTM and CNN . |
| Approach: | They propose a model that integrates pivot-based and NN modeling in a structure aware manner. |
| Outcome: | The proposed model can naturally feed structure aware text classifiers such as LSTM and CNN. |
Revisiting Simple Neural Probabilistic Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent advances in language modeling have been driven not only by advances in neural architectures, but also through hardware and optimization improvements. |
| Approach: | They revisit the neural probabilistic language model (NPLM) of Bengio et al. (2003) which simply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. |
| Outcome: | The proposed model performs better on word-level language model benchmarks than a baseline Transformer with short input contexts but struggles to handle long-term dependencies. |
Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study (D19-1)
Copied to clipboard
| Challenge: | Existing studies have focused on the ability of neural models to compute and employ phrase-level features attached to a set of words, such as subject number or whquestion words. |
| Approach: | They examine whether models can represent constituent-level features, using coordinated noun phrases as a case study. |
| Outcome: | The proposed model can combine gender and gender features to drive downstream expectations, while having less success with gender agreement. |