The Syntactic Acceptability Dataset (Preview): A Resource for Machine Learning and Linguistic Analysis of English (2024.lrec-main)
Copied to clipboard
| Challenge: | Syntactic acceptance dataset is a resource being designed for syntax and computational linguistics research. |
| Approach: | They propose to use the Syntactic Acceptability Dataset to examine the syntactical discourse. |
| Outcome: | The proposed dataset is the largest of its kind that is publicly accessible. |
Similar Papers
JCoLA: Japanese Corpus of Linguistic Acceptability (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural language models have exhibited outstanding performance in downstream tasks, yet there is limited understanding regarding the extent of their internalization of syntactic knowledge. |
| Approach: | They introduce a dataset that analyzes sentences annotated with binary acceptability judgments from linguistic textbooks and handbooks and splits them into in-domain and out-of-domain data. |
| Outcome: | The proposed datasets show that models can surpass human performance for in-domain data while no models can exceed human performance on out-of-domain datasets. |
Learning Syntactic Dense Embedding with Correlation Graph for Automatic Readability Assessment (2021.acl-long)
Copied to clipboard
| Challenge: | Existing deep learning models for automatic readability assessment discard linguistic features traditionally used for the task. |
| Approach: | They propose to incorporate linguistic features into machine learning models by learning syntactic dense embeddings based on linguistic feature extraction. |
| Outcome: | Experiments with six data sets of two proficiency levels show that the proposed model can perform better than existing models. |
How Well Do Text Embedding Models Understand Syntax? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text embedding models have not addressed syntactic understanding challenges, highlighting ineffectiveness and enhancing generalization ability. |
| Approach: | They propose to examine the ability of text embedding models to generalize across syntactic contexts. |
| Outcome: | The proposed models exhibit high similarity socres at this simple task. |
Do Neural Language Models Show Preferences for Syntactic Formalisms? (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work on interpretability of deep neural language models concludes that many properties of natural language syntax are encoded in their representational spaces. |
| Approach: | They propose to examine whether syntactic structure adheres to a surface-syntactical or deep syntaktic style of analysis. |
| Outcome: | The proposed model prefers Universal Dependencies (UD) over Surface-Syntactic Universal Dependency (SUD) with interesting variations across languages and layers. |
RuCoLA: Russian Corpus of Linguistic Acceptability (2022.emnlp-main)
Copied to clipboard
Vladislav Mikhailov, Tatiana Shamardina, Max Ryabinin, Alena Pestova, Ivan Smurov, Ekaterina Artemova
| Challenge: | Recent research has focused on evaluating the grammatical knowledge of language models with acceptability judgments. |
| Approach: | They propose to build a corpus of linguistic acceptability in Russian using a binary LA approach. |
| Outcome: | The proposed set of tests shows that the most widely used language models still fall behind humans by a large margin when detecting morphological and semantic errors. |
Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have examined the quality of labeled data in non-English languages. |
| Approach: | They annotate how datasets are created, input text and label sources, tools used to build them and what they study. |
| Outcome: | The results show that language-proficient NLP researchers' estimated availability correlates with dataset availability. |
Language model acceptability judgements are not always robust to context (2023.acl-long)
Copied to clipboard
Koustuv Sinha, Jon Gauthier, Aaron Mueller, Kanishka Misra, Keren Fuentes, Roger Levy, Adina Williams
| Challenge: | a recent study found that models prefer acceptable inputs over acceptable ones. |
| Approach: | They find that model judgements are generally robust when placed in randomly sampled linguistic contexts, but unstable when contexts match the test stimuli in syntactic structure. |
| Outcome: | The proposed model performance improves when contexts match syntactic structure, and declines when they are unacceptable. |
TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Whether politeness phenomena and strategies are universal across languages or not have been controversial among sociologists and linguists. |
| Approach: | They create a dataset containing three-way politeness annotations for 500 examples in each language, totaling 4.5K examples. |
| Outcome: | The proposed model shows a robust zero-shot transfer ability, but falls short of estimated human accuracy significantly. |
MELA: Multilingual Evaluation of Linguistic Acceptability (2024.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks on linguistic acceptability have been used to evaluate language models' ability to distinguish between acceptable and unacceptable sentences. |
| Approach: | They present the largest benchmark to date on linguistic acceptability: MELA . they establish LLM baselines on this benchmark and investigate cross-lingual transfer in acceptability judgements with XLM-R. |
| Outcome: | The proposed model outperforms open-source models on cross-lingual transfer in acceptability judgements. |
Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges (2025.acl-long)
Copied to clipboard
Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, Barbara Plank
| Challenge: | linguistics studies how context influences meaning of language and how people use it to convey implied meanings, emotions, and intentions. |
| Approach: | They analyze task designs, data collection methods, evaluation approaches and their relevance to real-world applications. |
| Outcome: | The findings highlight emerging trends, challenges, and gaps in existing benchmarks . the findings will contribute to more nuanced and context-aware NLP models . |