An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work attempts to explicitly incorporate human-defined linguistic priors into fine-tuning tasks. |
| Approach: | They replace parsed graphs or trees with trivial ones to investigate linguistic priors . they propose to use trivial graphs as baselines to design advanced knowledge fusion methods . |
| Outcome: | The use of trivial graphs improves performance in fully-supervised and few-shot settings. |
Similar Papers
Infusing Finetuning with Semantic Dependencies (2021.tacl-1)
Copied to clipboard
| Challenge: | Several diagnostics help to localize the benefits of our approach. |
| Approach: | They apply convolutional graph encoders to integrate semantic parses into task-specific finetuning. |
| Outcome: | The proposed approach yields benefits to natural language understanding (NLU) tasks in the GLUE benchmark. |
How transfer learning impacts linguistic knowledge in deep NLP models? (2021.findings-acl)
Copied to clipboard
| Challenge: | Several researchers have shown that deep NLP models learn non-trivial amount of linguistic knowledge, captured at different layers of the model. |
| Approach: | They propose to fine-tune pre-trained models towards downstream NLP tasks to capture linguistic knowledge. |
| Outcome: | The proposed model is adapted to GLUE tasks and retains linguistic information in the network while forgetting it. |
Linguistic Knowledge Can Enhance Encoder-Decoder Models (If You Let It) (2024.lrec-main)
Copied to clipboard
| Challenge: | a recent study has shown that pre-trained NLMs can capture syntax- and semantic-sensitive phenomena. |
| Approach: | They investigate whether fine-tuning pre-trained models with linguistic knowledge improves their performance in a target task. |
| Outcome: | The proposed enhancements improve models' performance in a target task, the authors show . the study includes models in Italian and English, and multilingual models in English and Italian . |
Disentangling Linguistic Relatedness from Task Alignment in Cross-Lingual Transfer (2026.acl-srw)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced natural language processing, yet their benefits remain concentrated in English and a small number of high-resource languages. |
| Approach: | They fine-tuned large language models (4B–671B parameters) on Arabic and evaluated zero-shot reading comprehension on Semitic languages and non-Semitic controls. |
| Outcome: | The results show that models with weak baselines improve across all languages, whereas strong-baseline models show only marginal gains regardless of language family. |
Are Visual-Linguistic Models Commonsense Knowledge Bases? (2022.coling-1)
Copied to clipboard
| Challenge: | PTLMs are used to extract knowledge from text on demand. |
| Approach: | They compare visual-linguistic and language-only visual-language models in a zero-shot commonsense question answering inference task. |
| Outcome: | The proposed models are highly promising on certain types of commonsense knowledge associated with the visual world. |
Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation (2021.eacl-main)
Copied to clipboard
| Challenge: | Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU). |
| Approach: | They empirically examine the usefulness of supervised parsing for semantic LU in LM-pretrained transformer networks. |
| Outcome: | The proposed model is based on LM-pretrained transformer networks with a biaffine parsing head and fine-tuned for LU tasks. |
CoLAKE: Contextualized Language and Knowledge Embedding (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models for integrating factual knowledge into pre-trained language models are shallow, static, and separately pre-train entities. |
| Approach: | They propose a method which integrates knowledge contexts from large-scale knowledge bases into a unified data structure. |
| Outcome: | The proposed model outperforms existing models on knowledge-driven tasks and knowledge probing tasks. |
On the Nature of BERT: Correlating Fine-Tuning and Linguistic Competence (2022.coling-1)
Copied to clipboard
| Challenge: | Several studies on the interpretation of Neural Language Models (NLMs) focus on the linguistic generalization abilities of pre-trained models, but little attention is paid to how the linguistic knowledge of the models changes during fine-tuning. |
| Approach: | They propose to examine whether a wide range of linguistic phenomena are forgotten during fine-tuning and whether it is possible to predict the fine- tuned accuracy solely relying on the assessed linguistic competence. |
| Outcome: | The proposed model can predict the evolution of written language competence of native language learners based on the assessed linguistic competence. |
Language Embeddings for Typology and Cross-lingual Transfer Learning (2021.acl-long)
Copied to clipboard
| Challenge: | Recent efforts to leverage multilingual datasets highlight potential of multilingual models that can perform well across various languages. |
| Approach: | They propose to generate language representations that capture relationships among languages and evaluate them using WALS and two extrinsic tasks. |
| Outcome: | The proposed model can be leveraged in cross-lingual tasks without parallel data . the proposed model is based on the World Atlas of Language Structures (WALS) and two extrinsic tasks . |
Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-Modal Knowledge Transfer (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models lack visual knowledge of common objects due to reporting bias. |
| Approach: | They investigate whether integrating visual knowledge into a language model can fill the gap . they use captions and images to transfer visual knowledge to 5 downstream tasks . |
| Outcome: | The proposed model can improve performance on 5 tasks that may need visual knowledge to solve the problem. |