Papers by Piotr Szymański
Is the Best Better? Bayesian Statistical Model Comparison for Natural Language Processing (2020.emnlp-main)
Copied to clipboard
| Challenge: | a recent study raises concerns about the use of standard splits to compare models . we compare the performance of six English part-of-speech taggers to those of other models based on standard split analysis . |
| Approach: | They propose a Bayesian statistical model comparison technique using k-fold cross-validation . they rank six English part-of-speech taggers across two data sets and three evaluation metrics . |
| Outcome: | The proposed method ranks English part-of-speech taggers on two data sets and three evaluation metrics. |
WER we are and WER we think we are (2020.findings-emnlp)
Copied to clipboard
Piotr Szymański, Piotr Żelasko, Mikolaj Morzy, Adrian Szymczak, Marzena Żyła-Hoppe, Joanna Banaszczak, Lukasz Augustyniak, Jan Mizgajski, Yishay Carmiel
| Challenge: | Recent reports of very low word error rates (WERs) achieved by modern automatic speech recognition systems are skepticism towards the accuracy of modern systems. |
| Approach: | They propose to use a dataset to test automatic speech recognition systems . they propose guidelines for creating real-life datasets with high quality annotations . |
| Outcome: | The proposed system achieves 81% of accuracy on human-chatbot interactions compared to the best reported results on human conversations and public benchmarks. |
Why Aren’t We NER Yet? Artifacts of ASR Errors in Named Entity Recognition in Spontaneous Speech Transcripts (2023.acl-long)
Copied to clipboard
Piotr Szymański, Lukasz Augustyniak, Mikolaj Morzy, Adrian Szymczak, Krzysztof Surdyk, Piotr Żelasko
| Challenge: | despite advances in language models, the transcript of spontaneous human-human conversations remains an insurmountable challenge for most models. |
| Approach: | They examine the relationship between ASR and NER errors which limit NER models' ability to recover entity mentions from spontaneous speech transcripts. |
| Outcome: | The proposed model fails even if no word errors are introduced by the ASR . the proposed model's performance deteriorates when applied to the ASL outputs . |