AdminSet and AdminBERT: a Dataset and a Pre-trained Language Model to Explore the Unstructured Maze of French Administrative Documents (2025.coling-main)
Copied to clipboard
| Challenge: | Pre-trained language models are used to analyze documents but administrative texts are unstructured and do not perform well. |
| Approach: | They propose a French pre-trained language model for the administrative domain . they compare it with a general domain language model and a large language model . |
| Outcome: | The proposed model improves performance on administrative and general domains. |
Similar Papers
Jargon: A Suite of Language Models and Evaluation Tasks for French Specialized Domains (2024.lrec-main)
Copied to clipboard
Vincent Segonne, Aidan Mannion, Laura Cristina Alonzo Canul, Alexandre Daniel Audibert, Xingyu Liu, Cécile Macaire, Adrien Pupier, Yongxin Zhou, Mathilde Aguiar, Felix E. Herron, Magali Norré, Massih R Amini, Pierrette Bouillon, Iris Eshkol-Taravella, Emmanuelle Esperança-Rodier, Thomas François, Lorraine Goeuriot, Jérôme Goulian, Mathieu Lafourcade, Benjamin Lecouteux, François Portet, Fabien Ringeval, Vincent Vandeghinste, Maximin Coavoux, Marco Dinarelli, Didier Schwab
| Challenge: | Pretrained language models are the de facto backbone of most state-of-the-art NLP systems. |
| Approach: | They propose a family of domain-specific pretrained PLMs for French focusing on three important domains: transcribed speech, medicine, and law. |
| Outcome: | The proposed models perform better on transcribed speech, medicine, and law domains than state-of-the-art models on a diverse set of tasks and datasets. |
DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains (2023.acl-long)
Copied to clipboard
Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, Pierre-Antoine Gourraud
| Challenge: | Recent studies have shown that pre-trained language models improve performance on a wide range of NLP tasks. |
| Approach: | They propose to use pre-trained language models to train medical domains on French language to compare performance with specialized ones. |
| Outcome: | The proposed models can take advantage of existing biomedical models in a foreign language by further pre-training them on our targeted data. |
FlauBERT: Unsupervised Language Model Pre-training for French (2020.lrec-1)
Copied to clipboard
Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, Didier Schwab
| Challenge: | Language models are a key step to achieve state-of-the-art results in many different Natural Language Processing (NLP) tasks. |
| Approach: | They propose to use a language model that is pre-trained on a large and heterogeneous French corpus to train continuous word representations. |
| Outcome: | The proposed model outperforms existing models on a large and heterogeneous French corpus. |
AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level (2022.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that large pre-trained language models are not sufficient for Hebrew. |
| Approach: | They propose a large pre-trained language model for Hebrew that recovers morphological segments encoded in contextualized embedding vectors. |
| Outcome: | The proposed model obtains state-of-the-art on all tasks beyond contemporary Hebrew baselines. |
LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish (2022.lrec-1)
Copied to clipboard
Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Lisa Veiber, Tegawende Bissyande, Jacques Klein, Andrey Boytsov, Clément Lefebvre, Anne Goujon
| Challenge: | Pre-trained Language Models such as BERT are ubiquitous in NLP but are scarce for low-resource languages such as Luxembourgish. |
| Approach: | They propose a BERT model for Luxembourgish language that they use to augment pre-training datasets by partially translating text data from a closely related language. |
| Outcome: | The proposed model outperforms the baseline model and the mBERT model in Luxembourgish. |
LaoPLM: Pre-trained Language Models for Lao (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) can capture different levels of concepts in context . previous work on Lao has been hampered by the lack of annotated datasets . |
| Approach: | They construct a text classification dataset to alleviate the resource-scarce situation of Lao . they evaluate them on two downstream tasks: part-of-speech tagging and text classification . |
| Outcome: | The proposed model can capture different levels of concepts in context and generate universal language representations. |
SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining (2021.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge-based PLMs are based on linked-entity information, but they only use linked-enemy information as auxiliary information. |
| Approach: | They propose to integrate semantic knowledge from neighbours of linked-entity into a medical PLM that integrates heterogeneous-entities into the homogeneously neighbouring entity structure. |
| Outcome: | Experiments show that SMedBERT outperforms baselines in knowledge-intensive Chinese medical tasks. |
Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing (2022.emnlp-main)
Copied to clipboard
Abbas Ghaddar, Yimeng Wu, Sunyam Bagga, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Xinyu Duan, Zhefeng Wang, Baoxing Huai, Xin Jiang, Qun Liu, Phillippe Langlais
| Challenge: | Existing pre-trained language models are not well-explored and are not reproducible in the literature. |
| Approach: | They propose to improve existing Arabic language pre-trained language models using a more methodical approach. |
| Outcome: | The proposed models outperform existing models on ALUE, a leaderboard-powered benchmark for Arabic NLU and NLG tasks. |
A Data-driven Approach to Named Entity Recognition for Early Modern French (2022.coling-1)
Copied to clipboard
| Challenge: | Named entity recognition is an important task in natural language processing. |
| Approach: | They propose to use a data-driven approach to identify historical French with fine-grained annotations instead of a specialised architecture to tackle particularities. |
| Outcome: | The proposed corpus is larger than the most popular NER evaluation corpora for both Contemporary English and French. |
Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Pre-trained language models provide the foundations for state-of-the-art performance across a wide range of natural language processing tasks, including text classification. |
| Approach: | They compare the performance of a linear classifier based on word embeddings with a pre-trained language model, i.e., BERT, across a wide range of datasets and classification tasks. |
| Outcome: | The proposed method outperforms baselines in standard datasets with large training sets, but in settings with small training datasets it performs better. |