| Challenge: | a growing interest in digital humanities for automatic processing and annotation of historical texts is generating new models for historical languages. |
| Approach: | They use POS-tagging and dependency parsing to evaluate contextual word embedding models . Old French is one of the historical languages for which they have the largest amount of syntactically annotated data . |
| Outcome: | The proposed model can be used to improve performance in Old French, the authors show . they use POS-tagging and dependency parsing to evaluate the model's quality . |
Similar Papers
Contextual Embeddings: When Are They Worth It? (2020.acl-main)
Copied to clipboard
| Challenge: | In recent years, rich contextual embeddings have enabled rapid progress on benchmarks like GLUE, but require significant computational resources during pretraining and during downstream task training and inference. |
| Approach: | They empirically compare contextual embeddings with classic pretrained embedders and a random word embeddable with a simple baseline. |
| Outcome: | The proposed models perform within 5 to 10% accuracy on industry-scale data. |
Give your Text Representation Models some Love: the Case for Basque (2020.lrec-1)
Copied to clipboard
Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, Eneko Agirre
| Challenge: | Word embeddings and pre-trained language models are expensive to train and are often used by small companies and research groups to build their own. |
| Approach: | They propose to use word embeddings and pre-trained language models to build rich representations of text and improve NLP tasks. |
| Outcome: | The proposed models perform better than publicly available versions in downstream NLP tasks for Basque. |
Building Static Embeddings from Contextual Ones: Is It Useful for Building Distributional Thesauri? (2022.lrec-1)
Copied to clipboard
| Challenge: | contextual language models are dominant in the field of Natural Language Processing, but they are not suitable for all uses. |
| Approach: | They propose a method for building word or type-level embeddings from contextual models . they evaluate a large set of English nouns from the perspective of extracting semantic similarity relations . |
| Outcome: | The proposed method can be used to build word or type embeddings from contextual models . it can be exploited for a wide set of English nouns, showing it can improve distributional thesauri . |
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020.acl-main)
Copied to clipboard
| Challenge: | a recent trend in neural NLP has been the introduction of feature-based and fine-tuning methods . we train monolingual contextualized word embeddings for five mid-resource languages . |
| Approach: | They use common Crawl corpus to train monolingual contextualized word embeddings . they compare performance of OSCAR-based and Wikipedia-based embeddables on part-of-speech tasks . |
| Outcome: | The results show that OSCAR-based and Wikipedia-based embeddings perform better than Wikipedia-style embedders on part-of-speech tagging and parsing tasks. |
On the Sentence Embeddings from Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained contextual representations like BERT have been widely used for NLP tasks. |
| Approach: | They propose to transform anisotropic sentence embedding distribution to smooth and isotropic Gaussian distribution by normalizing flows that are learned with an unsupervised objective. |
| Outcome: | The proposed method achieves significant performance gains over state-of-the-art embeddings on a variety of semantic textual similarity tasks. |
Static Embeddings as Efficient Knowledge Bases? (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent research investigates factual knowledge stored in large pretrained language models . masked sentences such as “Paris is the capital of [MASK]” are used as probes . |
| Approach: | They use masked sentences to test whether a language model can capture factual knowledge . they show that static embeddings perform better than PLMs when restricted to a candidate set . |
| Outcome: | The results show that static embeddings perform better than PLMs when restricted to a candidate set . |
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (D19-1)
Copied to clipboard
| Challenge: | Existing word embeddings were static, requiring all senses of a polysemous word to share the same representation. |
| Approach: | They found that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. |
| Outcome: | The results show that the representations of all words are not isotropic in any layer of the contextualizing model. |
Spying on Your Neighbors: Fine-grained Probing of Contextual Embeddings for Information about Surrounding Words (2020.acl-main)
Copied to clipboard
| Challenge: | a suite of probing tasks test contextual embeddings for encoding of information about surrounding words . authors: little is known about what information embeddables encode about the context words encode . a recent study shows that contextual embeds can be powerful for many tasks . |
| Approach: | They propose probing tasks that enable fine-grained testing of contextual embeddings . they examine popular contextual encoders and find that each encodes contextual information across tokens a little different . |
| Outcome: | The proposed probing tasks show that word embeddings encode information about words . the tests show that the encoded information is encoded across tokens with near-perfect recoverability . |
CamemBERT: a Tasty French Language Model (2020.acl-main)
Copied to clipboard
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, Benoît Sagot
| Challenge: | Pretrained language models are now ubiquitous in Natural Language Processing, but their use in other languages is limited. |
| Approach: | They propose to train monolingual Transformer-based model for other languages using web crawled data instead of Wikipedia data and a relatively small web crawl dataset leads to better results. |
| Outcome: | The proposed model performs as well as those obtained using larger datasets. |
BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to improve word embeddings for rare words are limited to uncontextualized word embeds. |
| Approach: | They propose a powerful architecture that can infer high-quality embeddings for rare words . they use the surface form and contexts of a word to interact in a deep architecture . |
| Outcome: | The proposed architecture can infer high-quality embeddings for rare words that are suitable as input representations for deep language models. |