Papers by Sebastian Schuster
When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing theories on how humans track discourse entities are based on the idea that humans maintain explicit memory representations for each entity that encode all properties of an entity and its relation to other entities. |
| Approach: | They adapt the psycholinguistic assessment of language models paradigm to higher-level linguistic phenomena and introduce an English evaluation suite that targets the knowledge of the interactions between sentential operators and indefinite NPs. |
| Outcome: | The evaluation suite targets the knowledge of the interactions between sentential operators and indefinite NPs and the models are challenged by multiple NP's and their behavior is not systematic. |
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Sequence-to-sequence models fail to generalize in hierarchy-sensitive manner when performing syntactic transformations. |
| Approach: | They evaluate whether seq2seq models generalize hierarchically on two transformations . they use pre-trained models and their multilingual variants to test their generalization . |
| Outcome: | The proposed models generalize hierarchically on two transformations in English and German. |
Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection (2020.lrec-1)
Copied to clipboard
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, Daniel Zeman
| Challenge: | Universal Dependencies is an open community effort to create cross-linguistically consistent treebank annotation for many languages. |
| Approach: | They describe version 2 of the universal guidelines and discuss major changes from UD v1 to UD 2 . they propose a morphological layer, a syntactic layer and a word segmentation layer . |
| Outcome: | The proposed treebanks are available for 90 languages and have been updated to meet the needs of multilingual parsers and researchers. |
Entity Tracking in Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies on the ability of large language models to track discourse entities have not been conducted. |
| Approach: | They propose to investigate whether large language models can track entities . they first investigate whether Flan-T5, GPT-3 and GPT-3.5 can track the state of entities based on an English description of the initial state and a series of state-changing operations. |
| Outcome: | The proposed task investigates whether language models can track entities based on language descriptions and state-changing operations. |
RExBench: Can coding agents autonomously implement AI research extensions? (2026.acl-long)
Copied to clipboard
| Challenge: | Existing large language model (LLM) agents are not capable of performing research extension tasks autonomously. |
| Approach: | They propose a benchmark to evaluate LLM agents' ability to extend existing AI research . they use extensions of 12 recently published research papers accompanied by domain expert-written instructions . |
| Outcome: | The proposed benchmark evaluates 12 LLM agents implemented using aider and OpenHands. |
Sentences with Gapping: Parsing and Reconstructing Elided Predicates (N18-1)
Copied to clipboard
| Challenge: | Sentences with gapping lack an overt predicate to indicate the relation between two or more arguments. |
| Approach: | They propose two methods for parsing to a Universal Dependencies graph representation that explicitly encodes the elided material with additional nodes and edges. |
| Outcome: | The proposed methods reconstruct elided material from dependency trees with high accuracy when the parser correctly predicts the existence of a gap. |
SpreadNaLa: A Naturalistic Code Generation Evaluation Dataset of Spreadsheet Formulas (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing datasets primarily target the use of code generation models to aid expert programmers in writing code. |
| Approach: | They propose a natural language code generation model that can translate English descriptions to spreadsheet formulas that can be used to do everyday data processing tasks. |
| Outcome: | The proposed model performs best among the evaluated methods but generates formulas that differ from human-generated ones. |
Harnessing the linguistic signal to predict scalar inferences (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Bayesian game-theoretic models of pragmatic reasoning can predict the strength of scalar inferences by using linguistic features. |
| Approach: | They propose to use a sentence encoder to predict the strength of scalar inferences by using a corpus of linguistic data. |
| Outcome: | The proposed model infers previously established associations between linguistic features and inference strength, suggesting that it learns to use linguistic feature to predict pragmatic inferences. |
Crowdsourcing a Large Corpus of Clickbait on Twitter (C18-1)
Copied to clipboard
Martin Potthast, Tim Gollub, Kristof Komlossy, Sebastian Schuster, Matti Wiegmann, Erika Patricia Garces Fernandez, Matthias Hagen, Benno Stein
| Challenge: | Clickbait is a nuisance on social media. |
| Approach: | a corpus of 38,517 annotated Twitter tweets was constructed to detect clickbait . the corpus was annotating tweets on 4-point scale by five annotators at Amazon's Mechanical Turk . |
| Outcome: | The corpus of 38,517 annotated Twitter tweets was used to evaluate 12 clickbait detectors submitted to the Clickbait Challenge 2017 . |
Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog (N19-1)
Copied to clipboard
| Challenge: | a lack of multilingual training data has hindered development of conversational AI models for task-oriented tasks . a new data set of 57k annotated utterances in english, spanish, and Thai is used to evaluate cross-lingual methods . |
| Approach: | They present a data set of 57k annotated utterances in English, Spanish and Thai . they evaluate three different cross-lingual transfer methods to identify user intents and slots . |
| Outcome: | The proposed model outperforms existing methods in English, Spanish and Thai . the proposed model is based on training data from three languages . |
Expectations over Unspoken Alternatives Predict Pragmatic Inferences (2023.tacl-1)
Copied to clipboard
| Challenge: | Scalar inferences (SI) are a signature example of how humans interpret language based on unspoken alternatives. |
| Approach: | They propose to use context-driven expectations to explain scale-based inferences . they find that expectedness of a strong scalemate captures SI rates within and across scales - but only under meaning-based view of alternatives. |
| Outcome: | The proposed model captures SI rates by expectedness of a strong scalemate as an alternative, but only under a meaning-based view of alternatives. |
SIGA: A Naturalistic NLI Dataset of English Scalar Implicatures with Gradable Adjectives (2024.lrec-main)
Copied to clipboard
| Challenge: | scalar implicatures are a phenomenon by which a speaker conveys the negation of a more informative utterance by producing a less informative . |
| Approach: | They propose to use a dataset to investigate the ability of language models to interpret utterances with scalar implicatures. |
| Outcome: | The proposed models perform significantly worse on in-domain and out-of-domain examples than other types of NLI examples. |