Papers by Tal Schuster
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | despite the importance of question answering, evaluations of QA systems are typically limited by manual annotations . despite this, little progress has been made in QA evaluations based on a single answer . |
| Approach: | They propose to extend over exact match (EM) with predefined rules or token-level F1 measure . they propose to use a BERT matching measure to approximate QA predictions . |
| Outcome: | The proposed model improves AE approximations and more accurately reflects the performance of systems. |
LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction (2023.acl-long)
Copied to clipboard
Jeremiah Milbauer, Annie Louis, Mohammad Javad Hosseini, Alex Fabrikant, Donald Metzler, Tal Schuster
| Challenge: | In many NLP tasks, the input text can be seen as a sequence of related segments. |
| Approach: | They propose a layer-adjustable interactions framework that contextualizes token representations by attending to all other tokens at each layer, leading to quadratic increase in compute effort with the input length. |
| Outcome: | The proposed model reduces 30-50% of attention FLOPs while maintaining high accuracy. |
When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing theories on how humans track discourse entities are based on the idea that humans maintain explicit memory representations for each entity that encode all properties of an entity and its relation to other entities. |
| Approach: | They adapt the psycholinguistic assessment of language models paradigm to higher-level linguistic phenomena and introduce an English evaluation suite that targets the knowledge of the interactions between sentential operators and indefinite NPs. |
| Outcome: | The evaluation suite targets the knowledge of the interactions between sentential operators and indefinite NPs and the models are challenged by multiple NP's and their behavior is not systematic. |
Consistent Accelerated Inference via Confident Adaptive Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Amortized or approximate computational methods increase efficiency, but can result in unpredictable performance costs. |
| Approach: | They propose a method that increases computational efficiency while guaranteeing a specifiable degree of consistency with the original model with high confidence. |
| Outcome: | The proposed method improves on four classification and regression tasks and can be used to predict the performance of the proposed model. |
Towards Debiasing Fact Verification Models (D19-1)
Copied to clipboard
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, Regina Barzilay
| Challenge: | Prior research has shown that data collection methods that use crowdsourcing introduce idiosyncratic biases that impact performance in unexpected ways. |
| Approach: | They propose a method to regularize the training data to avoid idiosyncrasies in the datasets that are used for fact verification. |
| Outcome: | The proposed model outperforms the existing model on the FEVER dataset, achieving 61.7% of the baseline. |
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Sequence-to-sequence models fail to generalize in hierarchy-sensitive manner when performing syntactic transformations. |
| Approach: | They evaluate whether seq2seq models generalize hierarchically on two transformations . they use pre-trained models and their multilingual variants to test their generalization . |
| Outcome: | The proposed models generalize hierarchically on two transformations in English and German. |
Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing (N19-1)
Copied to clipboard
| Challenge: | Existing methods for multilingual transfer are limited by their dynamic nature. |
| Approach: | They propose a method that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. |
| Outcome: | The proposed method outperforms the state-of-the-art on 6 languages, yielding an improvement of 6.8 LAS points on average. |
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence (2021.naacl-main)
Copied to clipboard
| Challenge: | Typical fact verification models use retrieved written evidence to verify claims . evidence sources change over time as more information is gathered and revised . a new benchmark for fact verification is VitaminC, which is contrastive in nature . |
| Approach: | They propose a benchmark that uses Wikipedia revisions to train models to discern and adjust to slight factual changes. |
| Outcome: | The proposed model improves accuracy by 10% on adversarial fact verification and 6% on adversary natural language inference. |
SDOH-NLI: a Dataset for Inferring Social Determinants of Health from Clinical Notes (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Social and behavioral determinants of health (SDOH) play a significant role in shaping health outcomes, and extracting these determinant from clinical notes is a first step to help healthcare providers systematically identify opportunities to provide appropriate care and address disparities. |
| Approach: | They propose a dataset that extracts social and behavioral determinants from clinical notes and uses them to form a natural language inference task. |
| Outcome: | The proposed dataset is based on publicly available notes and is more challenging than standard NLI benchmarks. |
On Reference (In-)Determinacy in Natural Language Inference (2025.findings-naacl)
Copied to clipboard
Sihao Chen, Chaitanya Malaviya, Alex Fabrikant, Hagai Taitelbaum, Tal Schuster, Senaka Buthpitiya, Dan Roth
| Challenge: | Using reference determinacy, models fail to recognize context mismatch in NLI examples . |
| Approach: | They propose a benchmark to identify reference ambiguity in NLI examples . they propose RD as a possible assumption for natural language inference . |
| Outcome: | The proposed benchmark identifies reference ambiguity in natural language inference examples . 80% false contradiction and >50% entailment predictions are found . |
PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing systems for Natural Language Inference (NLI) only recognize textual entailment relations on sentence-level . however, even a simple sentence often contains multiple propositions, i.e. distinct units of meaning conveyed by the sentence . |
| Approach: | They propose a system to recognize whether one text is textually entailed by another . they use a corpus of over 45K propositions annotated by human raters to study the textual entailment relation of each proposition in a sentence individually. |
| Outcome: | The proposed dataset can be used to understand the compositionality of NLI labels. |
SEMQA: Semi-Extractive Multi-Source Question Answering (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent proposed long-form question answering systems have shown promising capabilities, but attributing and verifying their generated abstractive answers can be difficult. |
| Approach: | They propose a task that summarises multiple sources in a semi-extractive fashion . they create a dataset with human-written semi-extractive answers to natural and generated questions . |
| Outcome: | The proposed task summarizes multiple sources in a semi-extractive fashion and produces fine in-line attributions by-design that are easy to verify, interpret, and evaluate. |
Attribute First, then Generate: Locally-attributable Grounded Text Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Recent efforts to address hallucinations in Large Language Models have focused on attributed text generation, which supplements generated texts with citations of supporting sources for post-generation fact-checking and corrections. |
| Approach: | They propose a locally-attributable text generation approach prioritizing concise attributions by identifying relevant source segments and conditioning the generation process on them. |
| Outcome: | The proposed method yields more concise citations than baselines and significantly reduces time required for fact verification by human assessors. |
Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in modeling and datasets demonstrate promising performance for NLI. |
| Approach: | They explore the direct zero-shot applicability of NLI models to real applications . they analyze the robustness of models to longer and out-of-domain inputs . |
| Outcome: | The proposed models are robust to longer and out-of-domain inputs and can perform on full documents. |
Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts? (2025.findings-naacl)
Copied to clipboard
| Challenge: | a recent study shows that human understanding of text depends on general semantic concepts of words that are robust to their superficial forms. |
| Approach: | They evaluate the accuracy of multilingual multilingual language models based on subword-level semantics . they form "semantic tokens" by merging semantically similar subwords and embeddings based upon the results . |
| Outcome: | The proposed models are able to make predictions on multilingual tasks with different tokenizers and model sizes. |