Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent zero-shot evaluations have highlighted important limitations in the abilities of language models (LMs) to perform meaning extraction. |
| Approach: | They propose to use in-context examples and instructions to improve LMs' robustness in performing property inheritance. |
| Outcome: | The proposed model can perform non-trivial property inheritance on in-context examples and instructions, but it is inconsistent with the task. |
Similar Papers
Transparency Helps Reveal When Language Models Learn Meaning (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing language models are trained to optimize unsupervised objectives on text . despite their centrality, current models do not represent natural language semantics well . |
| Approach: | They show that autoregressive and masked language models learn to emulate semantic relations between expressions when context-dependent . they argue that a learner that has access to all Java code can never learn execution . |
| Outcome: | a new study shows that language models fail to represent natural language semantics well . the authors show that the model learning fails when denotations are changed to be context-dependent . |
Language Models Use Monotonicity to Assess NPI Licensing (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) have become powerful approximators of human language . fewer studies have been done on what kind of formal semantic features are encoded by LMs . |
| Approach: | They propose a series of experiments that investigate the semantic knowledge of language models . they use diagnostic classifiers, linguistic acceptability tasks and a ranking method to investigate the models' inner workings. |
| Outcome: | The proposed method can be applied to LMs trained on filtered corpora and gain stronger insights into their generalizations. |
Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domains (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are known to distill knowledge from vast datasets during the pretraining phase. |
| Approach: | They propose to use in-context concept definitions as prompts for zero-shot sentence labeling tasks. |
| Outcome: | The proposed model outperforms the leading open-source models and their proprietary APIs in most cases. |
Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints (2025.acl-short)
Copied to clipboard
| Challenge: | Semantic Parsing improves performance of smaller models, but it is unclear whether it extends similarly to large language models. |
| Approach: | They propose a prompting approach that embeds semantic hints within the prompt to improve LLM performance. |
| Outcome: | The proposed approach improves LLMs’ performance across various tasks, highlighting the potential of integrating semantic information to improve LLM capabilities. |
Negation, Coordination, and Quantifiers in Contextualized Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Recent work has focused on specific tasks and on the learning outcome. |
| Approach: | They propose to decouple the weaknesses from specific tasks and focus on the embeddings per se and their mode of learning. |
| Outcome: | The proposed model can learn semantic constraints and how the context impacts their embeddings. |
Modelling Commonsense Properties Using Pre-Trained Bi-Encoders (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-trained language models can capture commonsense properties that are rarely expressed in text. |
| Approach: | They propose to fine-tune language models to explicitly model commonsense properties . they train separate concept and property encoders on extracted hyponym-hypernym pairs and generic sentences . |
| Outcome: | The proposed model can capture commonsense properties with higher accuracy than human models . a new study shows that the model can model commonsensence properties with much higher accuracy . |
Language Models Struggle to Use Representations Learned In-Context (2026.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that large language models are capable of inducing rich representations of data that are seen in-context . a novel task, adaptive world modeling, shows that even the most performant LLMs cannot reliably leverage novel semantics defined in-constitut. |
| Approach: | They propose to use in-context representations to induce rich representations of data . they also propose to probe models using a novel task to enable flexible deployment . |
| Outcome: | The proposed model can use in-context representations to complete simple downstream tasks. |
In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax (2024.naacl-long)
Copied to clipboard
| Challenge: | In-context learning is a common method for teaching large language models new tasks . given labeled examples in the input context, the model learns to perform the task without weight updates. |
| Approach: | They examine whether models guided via ICL infer the underlying structure of the task defined by the context or rely on superficial heuristics that only generalize to identically distributed examples. |
| Outcome: | The proposed model generalizes syntactically or linearly on out-of-distribution examples . the proposed model is able to generalize better on pre-trained models . |
That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for translation of ambiguous text use context to disambiguate meaning . current models for MTs consistently translate English idioms literally, whereas LMs are context-aware . |
| Approach: | They use a dataset of 512 pairs of English sentences to study semantic ambiguities . they use literal and figurative idioms to disambiguate intended meaning . |
| Outcome: | The results show that current models translate English idioms literally, even when the context suggests a figurative interpretation. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |