Papers by Evgenii Tsymbalov
How Certain is Your Transformer? (2021.eacl-main)
Copied to clipboard
Artem Shelmanov, Evgenii Tsymbalov, Dmitri Puzyrev, Kirill Fedyanin, Alexander Panchenko, Maxim Panov
| Challenge: | Obtaining reliable uncertainty estimations for such neural networks (NNs) is challenging due to the huge number of parameters in these deep learning models. |
| Approach: | They propose to use Monte Carlo dropout to estimate uncertainty for Transformer-based models and construct inexpensive estimates using Determinantal Point Processes. |
| Outcome: | The proposed estimates improve the quality of detection of error-prone instances. |
Uncertainty Estimation of Transformer Predictions for Misclassification Detection (2022.acl-long)
Copied to clipboard
Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, Manvel Avetisian, Leonid Zhukov
| Challenge: | Uncertainty estimation (UE) of model predictions is crucial step for a variety of tasks such as active learning, misclassification detection, adversarial attack detection, etc. |
| Approach: | They propose to modify UE methods for Transformer models for misclassification detection in named entity recognition and text classification tasks to improve model expressiveness and computational performance. |
| Outcome: | The proposed methods outperform computationally intensive methods on misclassification detection tasks and are based on a large dataset of simulated datasets. |
Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification (2024.findings-acl)
Copied to clipboard
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
| Challenge: | Large language models are notorious for producing erroneous claims in their output. |
| Approach: | They propose a fact-checking and hallucination detection pipeline based on token-level uncertainty quantification that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use. |
| Outcome: | The proposed method can fact-check the atomic claims in the output of large language models. |
Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for hallucination detection for text-based LLMs do not capture audio-specific signals. |
| Approach: | They propose to capture pathological attention patterns associated with hallucination using four attention-derived metrics to train lightweight logistic regression classifiers. |
| Outcome: | The proposed approach outperforms baselines on in-domain data and generalises to out-of-domain ASR settings. |