Licon: A Diverse, Controllable and Challenging Linguistic Concept Learning Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for Concept Learning focus on visual information, but visual information cannot present abstract concepts exactly, which struggles the introduction of novel concepts related to known concepts. |
| Approach: | They propose a benchmark where concepts in diverse forms are defined by linguistic descriptions and an entailment-based concept learning method to model the relationship among concepts. |
| Outcome: | The proposed benchmark is based on the existing visual concepts learning benchmarks and will be released to the public soon. |
Similar Papers
SpeciaLex: A Benchmark for In-Context Specialized Lexicon Learning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Specialized lexicons are collections of words with associated constraints such as special definitions, specific roles, and intended target audiences. |
| Approach: | They propose a benchmark to evaluate a language model’s ability to follow specialized lexicon-based constraints across 18 diverse subtasks with 1,785 test instances covering core tasks of Checking, Identification, Rewriting, and Open Generation. |
| Outcome: | The proposed model can follow specialized lexicon-based constraints across 18 diverse subtasks with 1,785 test instances covering core tasks Checking, Identification, Rewriting, and Open Generation. |
Inspecting the concept knowledge graph encoded by modern language models (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are used to solve tasks such as summarization and information retrieval. |
| Approach: | They propose to use word embeddings, text generators, context encoders to extract underlying knowledge graphs of nine influential language models. |
| Outcome: | The proposed model is able to encode word embeddings, text generators, and context encoders, but suffers from several inaccuracies. |
ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models (2024.findings-acl)
Copied to clipboard
Yanan Wu, Jie Liu, Xingyuan Bu, Jiaheng Liu, Zhanhui Zhou, Yuanxing Zhang, Chenchen Zhang, ZhiqiBai ZhiqiBai, Haibin Chen, Tiezheng Ge, Wanli Ouyang, Wenbo Su, Bo Zheng
| Challenge: | ConceptMath evaluates concept-wise mathematical reasoning of Large Language Models (LLMs) Existing benchmarks that evaluate general mathematical reasoning with an average accuracy fail to probe the fine-grained failure modes of mathematical reasoning on specific datasets. |
| Approach: | They introduce a bilingual, fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models. |
| Outcome: | The proposed benchmarks evaluate concept-wise mathematical reasoning of Large Language Models with concept-based accuracies. |
Inference Helps PLMs’ Conceptual Understanding: Improving the Abstract Inference Ability with Hierarchical Conceptual Entailment Graphs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to abstract inference ignore the *polysemy* and *hierarchical nature of concepts* . prevailing approaches disregard how arguments might entail differently across various concept levels, thereby missing potential enlargement connections. |
| Approach: | They propose a framework that organizes arguments hierarchically and delves into entailment relations at diverse concept levels. |
| Outcome: | The proposed framework improves the model's generalization and reasoning prowess in natural language inference. |
LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts (2026.eacl-long)
Copied to clipboard
| Challenge: | LM-LEXICON is a definition modeling approach that integrates data clustering, semantic expert learning, and model merging. |
| Approach: | They propose a definition modeling approach that integrates data clustering, semantic expert learning, and model merging using a sparse mixture-of-experts architecture. |
| Outcome: | The proposed model outperforms existing methods on five widely used benchmarks and achieves a BLEU score of 7%. |
Learning Concept Abstractness Using Weak Supervision (D18-1)
Copied to clipboard
Ella Rabinovich, Benjamin Sznajder, Artem Spector, Ilya Shnayderman, Ranit Aharonov, David Konopnicki, Noam Slonim
| Challenge: | Existing methods for inferring abstractness of words and expressions without labeled data are limited and limited. |
| Approach: | They propose a weakly supervised approach for inferring the property of abstractness of words and expressions in the absence of labeled data. |
| Outcome: | The proposed approach obtains high correlation with human labels in the absence of labeled data. |
Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing evaluation methods do not provide insight into how well a language model captures distinct linguistic skills essential for language understanding and reasoning. |
| Approach: | They propose a new format of NLI benchmark for evaluation of broad-coverage linguistic phenomena using a set of datasets and an evaluation procedure for diagnosing how well a language model captures reasoning skills. |
| Outcome: | The proposed model can diagnose model behavior and verify model learning quality. |
Cross-lingual Semantic Representation for NLP with UCCA (2020.coling-tutorials)
Copied to clipboard
| Challenge: | introductory tutorial to UCCA, a symbolic meaning representation for semantic representations. |
| Approach: | This tutorial introduces UCCA, a cross-linguistically applicable framework for semantic representation . it will provide a detailed introduction to the UCca annotation guidelines, design philosophy and available resources . |
| Outcome: | The tutorial will provide a detailed introduction to the UCCA framework and compare it to other meaning representations. |
Cabbage Sweeter than Cake? Analysing the Potential of Large Language Models for Learning Conceptual Spaces (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conceptual spaces are constructed from a set of quality dimensions, which are usually learned from human judgements, which means that applications of conceptual spaces are limited to narrow domains. |
| Approach: | They propose to use Large Language Models to learn perceptually grounded representations by comparing them to larger models of the BERT family. |
| Outcome: | The proposed models outperform the largest model, despite being 2 to 3 orders of magnitude smaller. |
An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work attempts to explicitly incorporate human-defined linguistic priors into fine-tuning tasks. |
| Approach: | They replace parsed graphs or trees with trivial ones to investigate linguistic priors . they propose to use trivial graphs as baselines to design advanced knowledge fusion methods . |
| Outcome: | The use of trivial graphs improves performance in fully-supervised and few-shot settings. |