| Challenge: | Neural language models (NLMs) encode lexical relations and syntactic structure, but their effectiveness is still unclear. |
| Approach: | They propose to use text as a model to model entities and situations as they evolve throughout a discourse. |
| Outcome: | The proposed models have functional similarities to linguistic models of dynamic semantics and can be learned with only text as training data. |
Similar Papers
Representations of Meaning in Neural Networks for NLP: a Thesis Proposal (2021.naacl-srw)
Copied to clipboard
| Challenge: | Neural networks are the state-of-the-art method of machine learning for many problems in NLP. |
| Approach: | They propose to examine the distribution of meaning in the vector space representation of words in neural networks trained for NLP tasks. |
| Outcome: | The proposed method would be compatible with distributional hypothesis, structuralism, and semantic holism. |
Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing models for linguistic representations of words are based on information extracted from large text corpora, and the sensory-motor experiences humans have with the world play an important role in determining word meaning. |
| Approach: | They propose to use contextualized word representations to learn semantic representations of words that align with human semantic intuitions. |
| Outcome: | The proposed models are shown to be more efficient on concrete word pairs than on abstract ones. |
Do Neural Language Models Inferentially Compose Concepts the Way Humans Can? (2024.lrec-main)
Copied to clipboard
| Challenge: | a new study shows that language models and humans may rely on different approaches to represent and compose lexical items across sentence structure. |
| Approach: | They propose to use a dataset to test the performance of neural language models and humans on inferentially driven conceptual compositions. |
| Outcome: | The proposed model elicits probability estimates for a noun in a minimally composed phrase . RoBERTa, BERT-large, and GPT-2 exhibited the closest resemblance to human responses . |
Meaning Representations for Natural Languages: Design, Models and Applications (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | a tutorial reviews the design of common meaning representations and SoTA models for predicting meaning representation. |
| Approach: | This tutorial reviews the design of common meaning representations and SoTA models for predicting meaning representation. authors propose a cutting-edge, full-day tutorial for all stakeholders in the AI community. |
| Outcome: | This tutorial reviews the design of common meaning representations and SoTA models for predicting meaning representation models . it also reviews the applications of meaning representation in downstream NLP tasks and real-world applications . |
Pre-trained language model representations for language generation (N19-1)
Copied to clipboard
| Challenge: | Pre-trained language model representations have been successful in a wide range of language understanding tasks. |
| Approach: | They propose to use pre-trained language model representations to integrate them into sequence to sequence models and apply it to machine translation and abstractive summarization. |
| Outcome: | The proposed model is able to perform 5.3 BLEU in machine translation and 5.3 on the full text version of CNN/DailyMail. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020.acl-main)
Copied to clipboard
| Challenge: | a priori, large neural language models are described as understanding or capturing meaning on tasks that are ostensibly meaningsensitive. |
| Approach: | They argue that a system trained only on form has no way to learn meaning . they argue that this is due to a misunderstanding of the relationship between form and meaning - which is a misconception in NLP . |
| Outcome: | The proposed model can't learn meaning because it only uses form as training data, the authors argue . they argue that a clear understanding of the distinction between form and meaning will guide the field towards better science around natural language understanding. |
Neural language models as psycholinguistic subjects: Representations of syntactic state (N19-1)
Copied to clipboard
| Challenge: | a recent study examines the extent to which neural network language models reflect incremental representations of syntactic state . we examine neural network model behavior on sentences chosen to probe specific aspects of the learned representations . |
| Approach: | They employ experimental methodologies developed in psycholinguistics to study syntactic representation in the human mind. |
| Outcome: | The proposed models are trained on large datasets and only sensitive to subtle cues . the results raise questions about the accuracy of the models and their performance . |
A Survey of Meaning Representations – From Theory to Practical Utility (2024.naacl-long)
Copied to clipboard
| Challenge: | Symbolic meaning representations of natural language text have been studied since at least the 1960s . with the availability of large annotated corpora, the field has recently seen several new developments . |
| Approach: | They propose a framework for expressing meaning in natural language text using annotated corpora and a set of tools for machine learning. |
| Outcome: | The frameworks are based on a set of theoretical and practical problems and their applications. |
Putting Words in Context: LSTM Language Models and Lexical Ambiguity (P19-1)
Copied to clipboard
| Challenge: | In language, a word can contribute a very different meaning depending on the context . lexical ambiguity involves both morphosyntactic and semantic aspects . |
| Approach: | They propose a method to probe hidden representations for lexical and contextual information about words. |
| Outcome: | The proposed method shows that both types of information are represented to a large extent, but there is room for improvement for contextual information. |