Papers by Aleksandar Savkov
Correlations between Word Vector Sets (D19-1)
Copied to clipboard
| Challenge: | Similarity measures based on word embeddings are easily competing with deep learning and expert-engineered systems on unsupervised semantic textual similarity tasks. |
| Approach: | They propose a new approach to measure word embeddings using pooling operations and correlation coefficients instead of pooling . they also propose centered kernel alignment as a natural generalisation of squared cosine similarity for sets of word vectors. |
| Outcome: | The proposed approach outperforms most recent methods while being much faster and trivial to implement. |
Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation (2022.emnlp-industry)
Copied to clipboard
Aleksandar Savkov, Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Anya Belz, Ehud Reiter
| Challenge: | Existing studies have shown that note generation is difficult due to subjective nature of many aspects of output quality. |
| Approach: | They propose a protocol that aims to increase objectivity by grounding evaluations in Consultation Checklists, which are created in a preliminary step and then used as a common point of reference during quality assessment. |
| Outcome: | The proposed protocol shows that the evaluations produced in the study are more objective than the original human note. |
Estimating Mutual Information Between Dense Word Embeddings (2020.acl-main)
Copied to clipboard
| Challenge: | Word embedding-based similarity measures are among the top-performing methods on unsupervised semantic textual similarity tasks. |
| Approach: | They propose a word embedding similarity measure that is a viable alternative to correlations and gives an excellent signal that correlates well with human judgements of similarity. |
| Outcome: | The proposed model outperforms existing state-of-the-art methods on hard NLP tasks. |
Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation (2022.acl-long)
Copied to clipboard
Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Damir Juric, Jack Flann, Ehud Reiter, Anya Belz, Aleksandar Savkov
| Challenge: | Recent studies suggest that note generation systems can be used to generate clinical consultation notes from the verbatim transcript of the consultation. |
| Approach: | They propose to use machine learning to generate consultation notes from the verbatim transcript of the consultation to evaluate their effectiveness. |
| Outcome: | The proposed model performs better than common model-based metrics like BertScore and is open-sourced. |
User-Driven Research of Medical Note Generation Software (2022.naacl-main)
Copied to clipboard
Tom Knoll, Francesco Moramarco, Alex Papadopoulos Korfiatis, Rachel Young, Claudia Ruffini, Mark Perera, Christian Perstl, Ehud Reiter, Anya Belz, Aleksandar Savkov
| Challenge: | Existing studies on how NLP systems could be used in clinical practice focus on technical difficulties and usability challenges involved in implementing them. |
| Approach: | They propose to use Speech Recognition to transcribe the audio of a medical consultation and then to train sequence-to-sequence models to summarise the transcript into a consultation note. |
| Outcome: | The proposed system generates notes in real time during a doctor-patient consultation and is able to capture the salient points of a consultation . the proposed system is based on three rounds of user studies in a live telehealth clinic and identifies a number of clinical use cases that could prove challenging for the system. |
PriMock57: A Dataset Of Primary Care Mock Consultations (2022.acl-short)
Copied to clipboard
| Challenge: | Recent advances in Automatic Speech Recognition (ASR) have made it possible to reliably produce automatic transcripts of clinician-patient conversations. |
| Approach: | They present a public access, high quality dataset of 57 mocked primary care consultations . they aim to offer a benchmark for conversational medical ASR and consultation note generation from transcripts. |
| Outcome: | The proposed dataset can be used as a benchmark for conversational medical ASR and consultation note generation from transcripts. |
Correlation Coefficients and Semantic Textual Similarity (N19-1)
Copied to clipboard
| Challenge: | Existing research into semantic textual similarity has focused on word embeddings . little attention has been devoted to similarity measures between word embeds - a new study shows . |
| Approach: | They show that cosine similarity is essentially equivalent to the Pearson correlation coefficient for all common word vectors. |
| Outcome: | The proposed model outperforms the existing model on word-level and sentence-level similarity benchmarks. |