Downstream Task Performance of BERT Models Pre-Trained Using Automatically De-Identified Clinical Data (2022.lrec-1)
Copied to clipboard
| Challenge: | Automatic de-identification systems introduce errors due to their imperfect precision and may negatively impact the utility of the de-identified dataset. |
| Approach: | They propose to de-identifie a large clinical corpus in Swedish by removing entire sentences containing sensitive data or by replacing sensitive words with realistic surrogates. |
| Outcome: | The proposed models are safe to distribute to other academic researchers and reduce privacy risks. |
Similar Papers
Data-Constrained Synthesis of Training Data for De-Identification (2025.acl-long)
Copied to clipboard
| Challenge: | sensitive domains lack widely available datasets due to privacy risks . recent studies have focused on evaluating the privacy of the synthetic text . |
| Approach: | They domain-adapt LLMs to clinical domain and generate synthetic clinical texts . they then generate NER models that can be annotated with tags for PII . |
| Outcome: | The proposed model performs better than the original model using smaller datasets. |
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretraining large (masked) language models over EHR data has yielded consistent performance gains across tasks. |
| Approach: | They propose to use large Transformers to release pretraining models over EHRs . they propose to recover patient names and conditions associated with them . |
| Outcome: | The proposed models recover patient names and conditions associated with patients . the proposed models share the model parameters for use by other researchers . |
Sensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERT (2020.lrec-1)
Copied to clipboard
| Challenge: | Massive digital data processing can endanger personal data privacy . anonymisation involves removing or replacing sensitive information from data . |
| Approach: | They propose to use a BERT-based sequence labelling model to conduct an experiment on clinical datasets in Spanish. |
| Outcome: | The proposed model outperforms existing models on clinical datasets in Spanish and shows that it is highly competitive with other models. |
Towards Fair and Efficient De-identification: Quantifying the Efficiency and Generalizability of De-identification Approaches (2026.findings-eacl)
Copied to clipboard
| Challenge: | a recent study has not examined their generalizability between formats, cultures, and genders. |
| Approach: | They evaluate large language models (LLMs) and small LLMs at clinical de-identification . they show that smaller models achieve comparable performance while substantially reducing inference cost . |
| Outcome: | The proposed models outperform larger models in de-identification tasks with limited data . the models can be fine-tuned with limited datasets to outperformed larger models . |
Impact of Training Instance Selection on Domain-Specific Entity Extraction using BERT (2022.naacl-srw)
Copied to clipboard
| Challenge: | Named entity recognition (NER) tasks require a large number of training examples and handcrafted features. |
| Approach: | They propose to fine-tune pre-trained language models such as BERT to achieve up to 80% F1 when fine- tuned on only 70 training examples. |
| Outcome: | The proposed model achieves 80% F1 when fine-tuned on only 70 training examples, especially on biomedical domain. |
Evaluating Pretraining Strategies for Clinical BERT Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing generic language models in specialized domains may be sub-optimal due to domain differences. |
| Approach: | They propose various strategies for adapting a generic language model to the target domain and various forms of vocabulary modifications to fine-tune it. |
| Outcome: | The proposed strategies outperform a general-domain language model but little difference in performance between the models. |
Re-train or Train from Scratch? Comparing Pre-training Strategies of BERT in the Medical Domain (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent years have witnessed the widespread use of transfer learning techniques in Natural Language Processing (NLP) |
| Approach: | They train BERT models from scratch using many configurations involving general and medical corpora. |
| Outcome: | The initial corpus only has a weak influence when these are further pre-trained on a medical corpus. |
Closing the Gap: Joint De-Identification and Concept Extraction in the Clinical Domain (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies show that de-identification is effective in the clinical domain but not in the downstream tasks. |
| Approach: | They propose a stacked model with restricted access to privacy sensitive information and a multitask model to investigate the effect of de-identification on clinical concept extraction. |
| Outcome: | The proposed model is stacked with restricted access to privacy sensitive information and a multitask model. |
Incorporating medical knowledge in BERT for clinical relation extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are used for diverse NLP tasks such as Information Extraction, Sentiment Analysis and Question/Answering. |
| Approach: | They propose to add medical knowledge to pre-trained language models to facilitate clinical relation extraction using a large text corpus. |
| Outcome: | The proposed model outperforms the state-of-the-art systems on the benchmark i2b2/VA 2010 clinical relation extraction dataset. |
How Far Is Too Far? Studying the Effects of Domain Discrepancy on Masked Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained masked language models perform strongly on a wide variety of NLP tasks. |
| Approach: | They propose a mechanism to quantify the difference in domains between the pre-trained model and the task and partition it using a cloze task. |
| Outcome: | The proposed model performs better on openly available e-commerce datasets than the original model on scientific and biomedical datasets. |