DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains (2023.acl-long)
Copied to clipboard
Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, Pierre-Antoine Gourraud
| Challenge: | Recent studies have shown that pre-trained language models improve performance on a wide range of NLP tasks. |
| Approach: | They propose to use pre-trained language models to train medical domains on French language to compare performance with specialized ones. |
| Outcome: | The proposed models can take advantage of existing biomedical models in a foreign language by further pre-training them on our targeted data. |
Similar Papers
DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domain (2024.lrec-main)
Copied to clipboard
Yanis Labrak, Adrien Bazoge, Oumaima El Khettari, Mickael Rouvier, Pacome Constant Dit Beaufils, Natalia Grabar, Béatrice Daille, Solen Quiniou, Emmanuel Morin, Pierre-Antoine Gourraud, Richard Dufour
| Challenge: | Existing benchmarks for pre-trained language models are limited to only a few languages . a limited number of tasks are evaluated on non-standardized protocols . |
| Approach: | They propose to aggregate diverse downstream tasks into a benchmark to assess PLMs' qualities . they evaluate 8 pre-trained masked language models on general and biomedical-specific data . |
| Outcome: | The proposed benchmark assesses pre-trained language models on 20 diversified tasks. |
Jargon: A Suite of Language Models and Evaluation Tasks for French Specialized Domains (2024.lrec-main)
Copied to clipboard
Vincent Segonne, Aidan Mannion, Laura Cristina Alonzo Canul, Alexandre Daniel Audibert, Xingyu Liu, Cécile Macaire, Adrien Pupier, Yongxin Zhou, Mathilde Aguiar, Felix E. Herron, Magali Norré, Massih R Amini, Pierrette Bouillon, Iris Eshkol-Taravella, Emmanuelle Esperança-Rodier, Thomas François, Lorraine Goeuriot, Jérôme Goulian, Mathieu Lafourcade, Benjamin Lecouteux, François Portet, Fabien Ringeval, Vincent Vandeghinste, Maximin Coavoux, Marco Dinarelli, Didier Schwab
| Challenge: | Pretrained language models are the de facto backbone of most state-of-the-art NLP systems. |
| Approach: | They propose a family of domain-specific pretrained PLMs for French focusing on three important domains: transcribed speech, medicine, and law. |
| Outcome: | The proposed models perform better on transcribed speech, medicine, and law domains than state-of-the-art models on a diverse set of tasks and datasets. |
Comprehensive Study on German Language Models for Clinical and Biomedical Text Understanding (2024.lrec-main)
Copied to clipboard
Ahmad Idrissi-Yaghir, Amin Dada, Henning Schäfer, Kamyar Arzideh, Giulia Baldini, Jan Trienes, Max Hasin, Jeanette Bewersdorff, Cynthia S. Schmidt, Marie Bauer, Kaleb E. Smith, Jiang Bian, Yonghui Wu, Jörg Schlötterer, Torsten Zesch, Peter A. Horn, Christin Seifert, Felix Nensa, Jens Kleesiek, Christoph M. Friedrich
| Challenge: | Pre-trained language models can struggle in specialized domains such as medicine . existing generalpurpose pre-tried models can be used and refined through further pre-training on domainspecific unlabeled data. |
| Approach: | They pre-trained German medical language models on 2.4B tokens from translated public data and 3B token of German clinical data. |
| Outcome: | The proposed models outperform clinical models on various downstream tasks in germany . the authors show that continuous pre-training can match or exceed clinical models trained from scratch . |
Development of pre-trained language models for clinical NLP in Spanish (2023.eacl-srw)
Copied to clipboard
| Challenge: | Clinical natural language processing aims to tackle language and prediction tasks using text from medical practice, such as clinical notes, prescriptions, and discharge summaries. |
| Approach: | They propose to build a clinical corpus big enough to implement a functional PLM. |
| Outcome: | The proposed model will be able to handle language and prediction tasks using clinical text while using biomedical and general text. |
ClinicalT5: A Generative Language Model for Clinical Text (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent generative language models like BART and T5 are gaining popularity with their competitive performance on text generation and tasks cast as generative problems. |
| Approach: | They propose to build domain-specific PLMs through fine-tuning or pre-training from scratch over domain corpora. |
| Outcome: | The proposed model outperforms existing models on domain-specific tasks and compares favorably with its close baselines. |
A Benchmark Evaluation of Clinical Named Entity Recognition in French (2024.lrec-main)
Copied to clipboard
| Challenge: | Masked Language Models (MLMs) have shown strong performance on many NLP tasks. |
| Approach: | They evaluate masked language models for biomedical French on the task of clinical named entity recognition using gold-standard corpora. |
| Outcome: | The proposed model outperforms standard models on the task of clinical named entity recognition in biomedical French while remaining lighter than current models. |
TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling (2025.findings-emnlp)
Copied to clipboard
| Challenge: | TransBERT framework for pre-training language models using exclusively synthetically translated text is limited in specialized domains. |
| Approach: | They propose a framework for pre-training language models using exclusively synthetically translated text . they also introduce a scalable translation toolkit that leverages synthetically trained data . |
| Outcome: | The proposed framework can be used to train language models using synthetically translated text . transCorpus toolkit can be scalable to the life sciences domain in french . |
SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining (2021.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge-based PLMs are based on linked-entity information, but they only use linked-enemy information as auxiliary information. |
| Approach: | They propose to integrate semantic knowledge from neighbours of linked-entity into a medical PLM that integrates heterogeneous-entities into the homogeneously neighbouring entity structure. |
| Outcome: | Experiments show that SMedBERT outperforms baselines in knowledge-intensive Chinese medical tasks. |
ViHealthBERT: Pre-trained Language Models for Vietnamese in Health Text Mining (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent large-scale language models show remarkable achievements in key NLP tasks such as Question Answering and Text Summarization. |
| Approach: | They propose a domain-specific pre-trained Vietnamese language model that outperforms the general domain language models. |
| Outcome: | The proposed model outperforms the general domain language models in Vietnamese datasets while outperforming the general-domain language models. |
Incorporating medical knowledge in BERT for clinical relation extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are used for diverse NLP tasks such as Information Extraction, Sentiment Analysis and Question/Answering. |
| Approach: | They propose to add medical knowledge to pre-trained language models to facilitate clinical relation extraction using a large text corpus. |
| Outcome: | The proposed model outperforms the state-of-the-art systems on the benchmark i2b2/VA 2010 clinical relation extraction dataset. |