Papers with OSCAR
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020.acl-main)
Copied to clipboard
| Challenge: | a recent trend in neural NLP has been the introduction of feature-based and fine-tuning methods . we train monolingual contextualized word embeddings for five mid-resource languages . |
| Approach: | They use common Crawl corpus to train monolingual contextualized word embeddings . they compare performance of OSCAR-based and Wikipedia-based embeddables on part-of-speech tasks . |
| Outcome: | The results show that OSCAR-based and Wikipedia-based embeddings perform better than Wikipedia-style embedders on part-of-speech tagging and parsing tasks. |
PAGnol: An Extra-Large French Generative Model (2022.lrec-1)
Copied to clipboard
Julien Launay, E.l. Tommasone, Baptiste Pannier, François Boniface, Amélie Chatelain, Alessandro Cappelli, Iacopo Poli, Djamé Seddah
| Challenge: | a growing number of pre-trained language models are available in many different languages. |
| Approach: | They propose a French-language GPT model with scaling laws to train it efficiently . they evaluate the models on discriminative and generative tasks in French . |
| Outcome: | The proposed model trains with the same computational budget as CamemBERT, a model 13 times smaller. |
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing web crawling pipelines are used to collect large corpora raw data, but the main way to collect such data is through manual data extraction. |
| Approach: | They propose to use a web crawler to extract and classify data from a multilingual web corpus and an automated annotation pipeline to improve it. |
| Outcome: | The proposed version of OSCAR could be used to pre-train large generative language models and other applications in Natural Language Processing and Digital Humanities. |