Papers by Kirill Fedyanin
How Certain is Your Transformer? (2021.eacl-main)
Copied to clipboard
Artem Shelmanov, Evgenii Tsymbalov, Dmitri Puzyrev, Kirill Fedyanin, Alexander Panchenko, Maxim Panov
| Challenge: | Obtaining reliable uncertainty estimations for such neural networks (NNs) is challenging due to the huge number of parameters in these deep learning models. |
| Approach: | They propose to use Monte Carlo dropout to estimate uncertainty for Transformer-based models and construct inexpensive estimates using Determinantal Point Processes. |
| Outcome: | The proposed estimates improve the quality of detection of error-prone instances. |
LM-Polygraph: Uncertainty Estimation for Language Models (2023.emnlp-demo)
Copied to clipboard
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov
| Challenge: | Large language models often "hallucinate" i.e., fabricate facts without providing users an apparent means to discern the veracity of their statements. |
| Approach: | They propose a framework with implementations of state-of-the-art UE methods for LLMs with unified program interfaces in Python. |
| Outcome: | The proposed framework implements state-of-the-art UE methods for LLMs with unified program interfaces in Python and an extendable benchmark for consistent evaluation by researchers. |
Uncertainty Estimation of Transformer Predictions for Misclassification Detection (2022.acl-long)
Copied to clipboard
Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, Manvel Avetisian, Leonid Zhukov
| Challenge: | Uncertainty estimation (UE) of model predictions is crucial step for a variety of tasks such as active learning, misclassification detection, adversarial attack detection, etc. |
| Approach: | They propose to modify UE methods for Transformer models for misclassification detection in named entity recognition and text classification tasks to improve model expressiveness and computational performance. |
| Outcome: | The proposed methods outperform computationally intensive methods on misclassification detection tasks and are based on a large dataset of simulated datasets. |