Papers by Venelin Kovatchev
Can vectors read minds better than experts? Comparing data augmentation strategies for the automated scoring of children’s mindreading ability (2021.acl-long)
Copied to clipboard
| Challenge: | In-domain experts are recruited to reannotate augmented samples and determine to what extent each strategy preserves the original rating. |
| Approach: | They implement 7 different data augmentation strategies for the task of automatic scoring of children’s ability to understand others’ thoughts, feelings, and desires. |
| Outcome: | The data augmentation strategies outperform task-agnostic augmentations and automatic augmentation systems perform worst on the MIND-CA corpus. |
An Analysis of Natural Language Inference Benchmarks through the Lens of Negation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for natural language inference ignore negations and can make inferences that are difficult to make. |
| Approach: | They propose a new benchmark for natural language inference in which negation plays a critical role. |
| Outcome: | The proposed benchmarks show that negation plays a critical role in inference judgments. |
“What is on your mind?” Automated Scoring of Mindreading in Childhood and Early Adolescence (2020.coling-main)
Copied to clipboard
Venelin Kovatchev, Phillip Smith, Mark Lee, Imogen Grumley Traynor, Irene Luque Aguilera, Rory Devine
| Challenge: | Existing studies show that children who excel at mindreading are more likely to be identified as popular by classmates and have reciprocated friendships. |
| Approach: | They propose to automate the scoring of mindreading ability in middle childhood and early adolescence using a new corpus of 11,311 question-answer pairs in English from 1,066 children aged from 7 to 14 . |
| Outcome: | The proposed scoring system is based on 11,311 question-answer pairs in English from 1,066 children aged from 7 to 14 . the results demonstrate the applicability of state-of-the-art NLP solutions to a new domain and task. |
Benchmark Transparency: Measuring the Impact of Data on Evaluation (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we quantify the impact that data distribution has on the performance and evaluation of NLP models. |
| Approach: | They propose to use disproportional stratified sampling to measure the data distribution across 6 different dimensions to quantify model performance. |
| Outcome: | The proposed framework measures the data distribution across 6 different dimensions and shows that it is statistically significant and predicts model performance. |
WARP-Text: a Web-Based Tool for Annotating Relationships between Pairs of Texts (C18-2)
Copied to clipboard
| Challenge: | Existing tools for annotating pairs of texts do not support detailed pairwise annotation. |
| Approach: | They present an open-source web-based tool for annotating relationships between pairs of texts . they propose to use WARP-Text to create multi-layer annotations and custom definitions . |
| Outcome: | The proposed tool can be used by project managers and annotators. |
Decomposing and Comparing Meaning Relations: Paraphrasing, Textual Entailment, Contradiction, and Specificity (2020.lrec-1)
Copied to clipboard
| Challenge: | SHARel is a new typology for decomposing and comparing multiple meaning relations . it consists of 26 linguistic and 8 reason-based categories and can be applied to all relations with a high inter-annotator agreement. |
| Approach: | They propose a new typology that consists of 26 linguistic and 8 reason-based categories and propose SHARel for decomposing and comparing multiple meaning relations. |
| Outcome: | The proposed method can be applied to all relations with high inter-annotator agreement. |
InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples (2022.coling-1)
Copied to clipboard
| Challenge: | InferES is an original corpus for Natural Language Inference (NLI) in European Spanish . |
| Approach: | They propose to implement and analyze a corpus-creating strategy utilizing expert linguists and crowd workers to provide high-quality data and facilitate the systematic evaluation of automated systems. |
| Outcome: | The proposed model obtains 72.8% accuracy and performs moderately well on negation-based adversarial examples. |
ETPC - A Paraphrase Identification Corpus Annotated with Extended Paraphrase Typology and Negation (L18-1)
Copied to clipboard
| Challenge: | Extended Paraphrase Typology addresses limitations of existing typologies . extended typology provides better means for evaluation and error analysis . |
| Approach: | a new typology copes with non-paraphrase pairs in paraphrase identification corpora, a paper proposes . a large corpus annotated with atomic paraphrase types is the largest to date . |
| Outcome: | The Extended Paraphrase Typology (EPT) and the Extended Typology Paraphrase Corpus (ETPC) address practical limitations of existing paraphrase typologies. |
ProtoTEx: Explaining Model Decisions with Prototype Tensors (2022.acl-long)
Copied to clipboard
| Challenge: | Neural models for NLP have yielded significant gains in predictive accuracy across tasks. |
| Approach: | They propose a white-box NLP classification architecture based on prototype networks . they propose an interleaved training algorithm that faithfully explains model decisions . |
| Outcome: | The proposed model matches BART-large and exceeds BERTlarge on propaganda detection tasks. |