Papers with Paris
Can Language Models Be Specific? How? (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained language models have a preference for more specific answers . however, there may exist multiple answers for a query, while not all answers are equally specific. |
| Approach: | They propose to build a benchmark for specificity testing by forming masked token prediction tasks with prompts. |
| Outcome: | The proposed methods improve the specificity of pre-trained language models without additional training. |
Static Embeddings as Efficient Knowledge Bases? (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent research investigates factual knowledge stored in large pretrained language models . masked sentences such as “Paris is the capital of [MASK]” are used as probes . |
| Approach: | They use masked sentences to test whether a language model can capture factual knowledge . they show that static embeddings perform better than PLMs when restricted to a candidate set . |
| Outcome: | The results show that static embeddings perform better than PLMs when restricted to a candidate set . |
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent work has shown that monolingual English language models can be used as knowledge bases. |
| Approach: | They use mBERT to query monolingual English language models with masked sentences to test their hypothesis. |
| Outcome: | The proposed model can be used to answer fill-in-the-blank questions in 53 languages. |
Probing Relational Knowledge in Language Models via Word Analogies (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have focused on probing relational knowledge by filling the blanks in pre-defined prompts such as “The capital of France is —” but these are affected by the co-occurrence of target relation words and entities in the pre-training corpus. |
| Approach: | They extend probing methodologies by using analogical proportions as a proxy to probe relational knowledge in transformer-based PLMs without directly presenting the desired relation. |
| Outcome: | The proposed methods are extremely accurate at (1) and (2), but have room for improvement for (3). |
TESA: A Task in Entity Semantic Aggregation for Abstractive Summarization (2020.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization systems focus on paraphrasing and simplifying the source content, to the exclusion of such semantic abstraction capabilities. |
| Approach: | They propose a dataset and task to fine tune an abstractive summarization model to generate aggregations of 5.3K entities from a crowd-sourced dataset. |
| Outcome: | The proposed task and dataset show that the proposed model can generate aggregations at a semantic level, but that it is too complex to use. |