Papers by Gorka Urbizu
How Well Can BERT Learn the Grammar of an Agglutinative and Flexible-Order Language? The Case of Basque. (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural Language Models (NLMs) have demonstrated effectiveness in acquiring skills related to human language use. |
| Approach: | They hypothesize that languages with complex grammar present substantial challenges during the pre-training phase . they constructed a test set that measures grammatical knowledge of BERT models trained under various pre-training configurations using corpus size, model size, number of epochs, and lemmatization. |
| Outcome: | The proposed model is based on a student-based minimal pairs test set with a grammatically correct and an incorrect sentence. |
Scaling Laws for BERT in Low-Resource Settings (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models require huge training corpora, which is unobtainable for most NLP practitioners. |
| Approach: | They propose power-law formulas that relate model size, corpora size and computation power to find the optimal settings in advance given a fixed budget. |
| Outcome: | The proposed models perform better on MLM and NLU tasks on four languages of different linguistic characteristics. |
BasqueGLUE: A Natural Language Understanding Benchmark for Basque (2022.lrec-1)
Copied to clipboard
| Challenge: | Natural Language Understanding (NLU) benchmarks are costly to develop and language-dependent . basqueGLUE is the first benchmark for Basque, a less-resourced language . |
| Approach: | They propose a benchmark for Basque, a less-resourced language, using existing datasets. |
| Outcome: | The proposed benchmarks take into account a wide and diverse set of NLU tasks that require some form of language understanding beyond the detection of superficial clues. |
Not Enough Data to Pre-train Your Language Model? MT to the Rescue! (2023.findings-acl)
Copied to clipboard
| Challenge: | In recent years, transformer-based language models (LMs) have become the default approach for many NLP tasks. |
| Approach: | They compare the performance of transformer-based language models with machine-translated corpora. |
| Outcome: | The proposed model can be improved with real data, but further research is needed. |