Papers by Hinrich Schütze

Monolingual and Multilingual Reduction of Gender Bias in Contextualized Representations (2020.coling-main)

Challenge:	a novel topic model is proposed to allow topical trends to be captured in temporal collections of documents.
Approach:	They propose a novel unsupervised neural dynamic topic model where topics are influenced by topic discovery over time.
Outcome:	The proposed model shows better generalization, topic interpretation, evolution and trends compared to state-of-the-art models .

Copied to clipboard

Sheng Liang, Philipp Dufter, Hinrich Schütze

A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters (2021.acl-long)

Challenge:	Prior work identifies a linear gender subspace and removes gender information by eliminating the subspace.
Approach:	They propose to use DensRay to obtain interpretable dense subspaces by applying it to attention heads and layers of BERT.
Outcome:	The proposed method performs on-par with prior approaches, but is more robust and preserves language model performance better.

Copied to clipboard

Mengjie Zhao, Yi Zhu, Ehsan Shareghi, Ivan Vulić, Roi Reichart, Anna Korhonen, Hinrich Schütze

Predicting the Growth of Morphological Families from Social and Linguistic Factors (2020.acl-main)

Challenge:	Few-shot crosslingual transfer outperforms zero-shot with pretrained encoders like multilingual BERT.
Approach:	They conduct an experimental study on 40 sets of sampled few shots for six diverse NLP tasks across up to 40 languages.
Outcome:	The proposed model outperforms state-of-the-art approaches on lexical features and a full model finetuning approach outperformed several state- of-the art approaches.

Copied to clipboard

Valentin Hofmann, Janet Pierrehumbert, Hinrich Schütze

Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly (2020.acl-main)

Challenge:	a burst in token frequency of the word "trump" in social media before the 2016 presidential election is a prime indicator of topical dynamics.
Approach:	They propose a task of Morphological Family Expansion Prediction to predict the size of a morphological family by analyzing a reddit corpus.
Outcome:	The proposed task predicts the increase in the size of a morphological family on a reddit corpus.

Copied to clipboard

Nora Kassner, Hinrich Schütze

UCxn: Typologically-Informed Annotation of Constructions Atop Universal Dependencies (2024.lrec-main)

Challenge:	Petroni et al. ( 2019) show that PLMs can recall factual knowledge that is part of their training corpus.
Approach:	They propose two new probing tasks that analyze factual knowledge stored in pretrained language models (PLMs) they introduce negated and non-negated cloze questions and add misprimes to clozing questions to mimic human priming methods.
Outcome:	The proposed tasks outperform automatically extracted knowledge bases on QA.

Copied to clipboard

Leonie Weissweiler, Nina Böbel, Kirian Guiller, Santiago Herrera, Wesley Samuel Scivetti, Arthur Lorenzi, Nurit Melnik, Archna Bhatia, Hinrich Schütze, Lori Levin, Amir Zeldes, Joakim Nivre, William Croft, Nathan Schneider

BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA (2020.findings-emnlp)

Challenge:	Grammatical constructions that convey meaning through a particular combination of several morphosyntactic elements are not labeled holistically.
Approach:	They propose to augment UD annotations with a ‘UCxn’ annotation layer for such meaning-bearing grammatical constructions and to approach this in a typologically informed way so that morphosyntactic strategies can be compared across languages.
Outcome:	The proposed annotation layer could be used to annotate meaning-bearing constructions across languages and to compare them across languages.

Copied to clipboard

Nora Kassner, Hinrich Schütze

SilverAlign: MT-Based Silver Data Algorithm for Evaluating Word Alignment (2024.lrec-main)

Challenge:	Pretrained language models (PLMs) capture a diverse range of linguistic and factual knowledge without the use of finetuning.
Approach:	They combine BERT with a traditional information retrieval step and a kNN search to improve the recall of facts encountered during training.
Outcome:	The proposed approach outperforms baselines on cloze-style QA by large margins without further training.

Copied to clipboard

Abdullatif Koksal, Silvia Severini, Hinrich Schütze

Machine Translation

Challenge:	Word alignments are crucial for a variety of NLP tasks.
Approach:	They propose a method to automatically create silver data for evaluation of word aligners by exploiting machine translation and minimal pairs.
Outcome:	The proposed method correlates with gold benchmarks for 9 language pairs, making it a valid resource for evaluation of different languages and domains when gold data is not available.

Continuous Entailment Patterns for Lexical Inference in Context (2021.emnlp-main)

Copied to clipboard

Martin Schmitt, Hinrich Schütze

Multi-Multi-View Learning: Multilingual and Multi-Representation Entity Typing (D18-1)

Challenge:	Pretrained language models can be used to perform lexical inference in context tasks with relatively small training data.
Approach:	They propose to combine a pretrained language model with textual patterns to improve performance in both zero-shot and few-shot settings.
Outcome:	The proposed method compares pre-trained models with textual patterns on two established benchmarks for lexical inference in context (LIiC) the results show that the proposed patterns improve performance on LIiC, setting a new state of the art.

Copied to clipboard

Yadollah Yaghoobzadeh, Hinrich Schütze

Few-Shot Text Generation with Natural Language Instructions (2021.emnlp-main)

Challenge:	Accurate and complete knowledge bases (KBs) are paramount in NLP.
Approach:	They employ multiview learning for increasing the accuracy and coverage of entity type information in KBs by taking high- and low-resource languages from Wikipedia.
Outcome:	The proposed learning improves the accuracy and coverage of knowledge bases (KBs) by combining language and representation.

Copied to clipboard

Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021.tacl-1)

Challenge:	Existing approaches to text generation combine task descriptions and examples with supervised learning.
Approach:	They propose a method for text generation that is based on pattern-exploiting training.
Outcome:	The proposed approach improves on several summarization and headline generation datasets.

Copied to clipboard

Timo Schick, Sahana Udupa, Hinrich Schütze

Multi-source Neural Topic Modeling in Multi-view Embedding Spaces (2021.naacl-main)

Challenge:	Pretrained language models pick up and reproduce undesirable biases when trained on large, unfiltered crawls from the Internet.
Approach:	They propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text.
Outcome:	The proposed approach reduces the probability of a language model producing problematic text by giving only a textual description of the undesired behavior.

Copied to clipboard

Pankaj Gupta, Yatin Chaudhary, Hinrich Schütze

Does She Wink or Does She Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models (2021.eacl-main)

Challenge:	Recent work has used pre-trained word embeddings to address data sparsity in short-text or small document collections.
Approach:	They propose a neural topic modeling framework using multi-view embedding spaces to improve topic quality and deal with polysemy.
Outcome:	The proposed framework improves topic quality and deal with polysemy.

Copied to clipboard

Lutfi Kerem Senel, Hinrich Schütze

SherLIiC: A Typed Event-Focused Lexical Inference Benchmark for Evaluating Natural Language Inference (P19-1)

Challenge:	Existing probing datasets focus on knowledge about relations between words and entities.
Approach:	They propose a new probing task to evaluate word understanding directly using dictionary definitions of words.
Outcome:	The proposed task evaluates word understanding directly using dictionary definitions of words.

Copied to clipboard

Martin Schmitt, Hinrich Schütze

Reasoning

Challenge:	We evaluate a large number of strong baselines on SherLIiC, ranging from semantic vector space models to state of the art neural models of natural language inference (NLI).
Approach:	They propose a testbed for lexical inference in context consisting of 3985 manually annotated inference rule candidates and a set of 960k unlabeled InfCands.
Outcome:	The proposed testbed is based on 3985 manually annotated inference rule candidates (InfCands) and 190k typed textual relations between Freebase entities extracted from the large entity-linked corpus ClueWeb09.

End-Task Oriented Textual Entailment via Deep Explorations of Inter-Sentence Interactions (P18-2)

Copied to clipboard

Wenpeng Yin, Dan Roth, Hinrich Schütze

Natural Language Interfaces

Challenge:	Existing datasets for textual entailment (TE) have been used to study TE.
Approach:	They propose a deep explorations of inter-sentence interactions for textual entailment task that uses a convolution to make important words in P and H play a dominant role in learnt representations.
Outcome:	Experiments show that the pretrained DEISTE on SciTail gets 5% improvement over prior state of the art and that it generalizes well on RTE-5.

E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2020.findings-emnlp)

Copied to clipboard

Nina Poerner, Ulli Waltinger, Hinrich Schütze

LLM-driven Instruction Following: Progresses and Concerns (2023.emnlp-tutorial)

Challenge:	Existing methods to enhance BERT with factual knowledge about entities require no additional pretraining and no changes to the encoder itself.
Approach:	They propose a way to inject factual knowledge into the pretrained BERT model by aligning Wikipedia2Vec entity vectors with BERT's native wordpiece vector space and feeding the aligned entity vector into BERT as if they were wordpieces.
Outcome:	The proposed version outperforms baseline models on unsupervised question answering, supervised relation classification and entity linking tasks.

Copied to clipboard

Wenpeng Yin, Qinyuan Ye, Pengfei Liu, Xiang Ren, Hinrich Schütze

Challenge:	a tutorial on task instruction is aimed at researchers and practitioners interested in NLP generalization . labeled examples are unlikely to be available in large numbers or do not exist .
Approach:	This tutorial will examine the progress of natural language processing (NLP) using labeled examples. authors propose that task instructions act as a novel resource for supervision.
Outcome:	This tutorial aims to answer questions about instruction-driven NLP . it focuses on the use of task instructions in a low-shot scenario .

Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages (2023.findings-acl)

Copied to clipboard

Ercong Nie, Sheng Liang, Helmut Schmid, Hinrich Schütze

An Unsupervised Joint System for Text Generation from Knowledge Graphs and Semantic Parsing (2020.emnlp-main)

Challenge:	Multilingual pretrained language models (MPLMs) perform strongly in cross-lingual transfer.
Approach:	They propose to augment context with similar sentences retrieved from a high-resource language (HRL) they find a significant correlation between cross-lingual transfer performance and similarity between high- and low-resourced languages .
Outcome:	The proposed model outperforms finetuning by 3.7% on three downstream tasks with multilingual parallel test sets across 10 LRLs covering 6 language families in unlabeled and labeled settings.

Copied to clipboard

Martin Schmitt, Sahand Sharifzadeh, Volker Tresp, Hinrich Schütze

Joint Bootstrapping Machines for High Confidence Relation Extraction (N18-1)

Challenge:	Knowledge graphs (KGs) vary greatly from one domain to another, resulting in a lack of domain-specific parallel graph-text data.
Approach:	They propose an unsupervised approach to graph-to-text generation and text-to graph knowledge extraction using WebNLG v2.1 and a new benchmark leveraging scene graphs from Visual Genome.
Outcome:	The proposed approach outperforms baselines on WebNLG v2.1 and a new benchmark leveraging scene graphs from Visual Genome.

Copied to clipboard

Pankaj Gupta, Benjamin Roth, Hinrich Schütze

How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives (2023.acl-short)

Challenge:	Existing semi-supervised bootstrapping methods for relationship extraction lack labeled data.
Approach:	They propose a semi-supervised bootstrapping method that protects against semantic drift . they expand entities and templates in parallel and in mutually constraining fashion in each iteration .
Outcome:	Experimental results show that BREX improves on state-of-the-art methods for four relationships.

Copied to clipboard

Xinpeng Wang, Leonie Weissweiler, Hinrich Schütze, Barbara Plank

Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words (2021.acl-long)

Challenge:	Recent studies show that intermediate layer distillation (ILD) objectives improve model compression, but a comprehensive evaluation of distillation objectives in both task-specific and task-agnostic settings is lacking.
Approach:	They propose to use knowledge distillation to improve model compression by transferring knowledge from one model to another.
Outcome:	The proposed framework improves on the task of QNLI with lower teacher layers and higher teacher layers.

Copied to clipboard

Valentin Hofmann, Janet Pierrehumbert, Hinrich Schütze

Verbing Weirds Language (Models): Evaluation of English Zero-Derivation in Five LLMs (2024.lrec-main)

Challenge:	Pretrained language models (PLMs) are based on fixed-size vocabularies of words and subwords that are generated by compression algorithms such as bytepair encoding.
Approach:	They propose to use BERT as an example PLM to study its semantic representations of English derivatives to test their hypothesis.
Outcome:	The proposed model outperforms BERT on a series of semantic probing tasks.

Copied to clipboard

David R. Mortensen, Valentina Izrailevitch, Yunze Xiao, Hinrich Schütze, Leonie Weissweiler

Recurrent One-Hop Predictions for Reasoning over Knowledge Graphs (C18-1)

Challenge:	Lexical-syntactic flexibility is a hallmark of English morphology . conversion involves placing a word with one part of speech in a non-prototypical context .
Approach:	They propose to test lexical-syntactic flexibility in the form of conversion . conversion is a process where a word with one part of speech is placed in a non-prototypical context .
Outcome:	The proposed task tests the ability of five language models to generalize over words with a non-prototypical part of speech.

Copied to clipboard

Wenpeng Yin, Yadollah Yaghoobzadeh, Hinrich Schütze

Structured Data in NLP

Challenge:	Large scale knowledge graphs (KGs) such as Freebase are generally incomplete.
Approach:	They propose a model that predicts entities at each step of mh-KB paths . the model is based on recurrent neural networks and vector representations of entities and relations .
Outcome:	The proposed models show state-of-the-art for two important multi-hop KG reasoning tasks.

How Transliterations Improve Crosslingual Alignment (2025.coling-main)

Copied to clipboard

Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Ayyoob ImaniGooghari, Orgest Xhelili, Haotian Ye, Chunlan Ma, François Yvon, Hinrich Schütze

A Crosslingual Investigation of Conceptualization in 1335 Languages (2023.acl-long)

Challenge:	Recent studies show that post-aligning multilingual pretrained language models improve crosslingual alignment, but it is unclear how and why this is achieved.
Approach:	They propose to explicitly evaluate crosslingual alignment by adding transliterations to models using original and transliterated data.
Outcome:	The proposed approach improves crosslingual alignment even for random sentences.

Copied to clipboard

Yihong Liu, Haotian Ye, Leonie Weissweiler, Philipp Wicke, Renhao Pei, Robert Zangenfeind, Hinrich Schütze

Graph Algorithms for Multiparallel Word Alignment (2021.emnlp-main)

Challenge:	Conceptualizer is a method that creates a bipartite directed alignment graph between source language concepts and sets of target language strings.
Approach:	They propose a method that creates a bipartite directed alignment graph between source language concepts and sets of target language strings.
Outcome:	The proposed method has good alignment accuracy across all languages and on 32 Swadesh concepts.

Copied to clipboard

Ayyoob ImaniGooghari, Masoud Jalili Sabet, Lutfi Kerem Senel, Philipp Dufter, François Yvon, Hinrich Schütze

Measuring and Improving Consistency in Pretrained Language Models (2021.tacl-1)

Challenge:	Word alignments are useful for typological research and can be used in machine translation systems.
Approach:	They propose to exploit the multiparallelity of parallel corpora by representing bilingual alignments as a graph and then predicting additional edges.
Outcome:	The proposed algorithm improves the accuracy of bilingual alignments by 28% over baseline algorithms.

Copied to clipboard

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg

Static Embeddings as Efficient Knowledge Bases? (2021.naacl-main)

Challenge:	In this paper, we examine whether pretrained language models are consistent with factual knowledge.
Approach:	They propose a method to improve consistency of pretrained language models . consistency is a desirable property of a good language understanding model, they argue .
Outcome:	The proposed model improves consistency and shows that it is effective.

Copied to clipboard

Philipp Dufter, Nora Kassner, Hinrich Schütze

Linguistically Informed Relation Extraction and Neural Architectures for Nested Named Entity Recognition in BioNLP-OST 2019 (D19-57)

Challenge:	Recent research investigates factual knowledge stored in large pretrained language models . masked sentences such as “Paris is the capital of [MASK]” are used as probes .
Approach:	They use masked sentences to test whether a language model can capture factual knowledge . they show that static embeddings perform better than PLMs when restricted to a candidate set .
Outcome:	The results show that static embeddings perform better than PLMs when restricted to a candidate set .

Copied to clipboard

Pankaj Gupta, Usama Yaseen, Hinrich Schütze

Language Models for Lexical Inference in Context (2021.eacl-main)

Challenge:	Named Entity Recognition (NER) and Relation Extraction (RE) are essential tools in distilling knowledge from biomedical literature.
Approach:	They propose to use Named Entities to perform nested entities extraction, Entity Normalization and Relation Extraction to generalize the approach to different languages.
Outcome:	The proposed approach can be generalized to different languages and showed it’s effectiveness for English and Spanish text.

Copied to clipboard

Martin Schmitt, Hinrich Schütze

News Article Teaser Tweets and How to Generate Them (N19-1)

Challenge:	Lexical inference in context (LIiC) is a variant of the natural language inference task focused on lexical semantics.
Approach:	They propose three approaches based on pretrained language models for LIiC . they propose a few-shot NLI classifier and a relation induction approach based upon handcrafted patterns expressing the semantics of lexical inference.
Outcome:	The proposed approaches outperform the previous state of the art and show their potential for LIiC.

Copied to clipboard

Sanjeev Kumar Karn, Mark Buckley, Ulli Waltinger, Hinrich Schütze

Challenge:	A teaser is a short reading suggestion for an article that is illustrative and includes curiosity-arousing elements to entice potential readers to read particular news items.
Approach:	They propose a benchmark and baseline system for the process of generating teasers.
Outcome:	The proposed system is best performing with a pointer network.

BioNLP-OST 2019 RDoC Tasks: Multi-grain Neural Relevance Ranking Using Topics and Attention Based Query-Document-Sentence Interactions (D19-57)

Copied to clipboard

Pankaj Gupta, Yatin Chaudhary, Hinrich Schütze

Information Retrieval

Challenge:	Our best systems achieved 1st rank and scored 0.86 mAP and 0.58 macro average accuracy in Task-1 and Task-2 respectively.
Approach:	They propose to use attention-based supervised neural topic model and SVM for retrieval and ranking of PubMed abstracts and to use BM25 and other relevance measures for re-ranking.
Outcome:	The proposed system scored 0.86 mAP and 0.58 macro average accuracy in the RDoC Tasks of BioNLP-OST 2019 .

TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data (2025.coling-main)

Copied to clipboard

Yihong Liu, Chunlan Ma, Haotian Ye, Hinrich Schütze

Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity (2020.acl-main)

Challenge:	Existing mPLMs that handle non-transliterated data are not sufficient to train crosslingual models.
Approach:	They propose a framework to transliterate related languages into a common script by exploiting existing mPLMs and their tokenizer without any training.
Outcome:	The proposed framework can create strong baselines for data that is transliterated into a common script by exploiting an existing mPLM and its tokenizer without any training.

Copied to clipboard

Nina Poerner, Ulli Waltinger, Hinrich Schütze

Graph-Based Multilingual Label Propagation for Low-Resource Part-of-Speech Tagging (2022.emnlp-main)

Challenge:	Existing word embeddings combine complementary strengths of their components to achieve unsupervised semantic similarity (STS).
Approach:	They propose to ensemble pre-trained sentence encoders into sentence meta-embeddings to achieve unsupervised Semantic Textual Similarity (STS) they adapt dimensionality reduction, generalized Canonical Correlation Analysis and cross-view auto-encoders to their work.
Outcome:	The proposed method achieves 3.7% to 6.4% Pearson’s r over single-source word embeddings on the STS Benchmark and on the StS12-STS16 datasets.

Copied to clipboard

Ayyoob ImaniGooghari, Silvia Severini, Masoud Jalili Sabet, François Yvon, Hinrich Schütze

Erratum: Measuring and Improving Consistency in Pretrained Language Models (2021.tacl-1)

Challenge:	Part-of-Speech (POS) tagging is an important component of the NLP pipeline, but many low-resource languages lack labeled training data.
Approach:	They propose a method for transferring labels from high-resource sources to low-resourced target languages using a graph-based label propagation method.
Outcome:	The proposed method achieves state-of-the-art for unsupervised POS tagging of low-resource languages.

Copied to clipboard

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg

Automatic Domain Adaptation Outperforms Manual Domain Adaptation for Predicting Financial Outcomes (P19-1)

Challenge:	During production of this paper, an error was introduced to the formula on the bottom of the right column of page 1020.
Approach:	the formula was changed in the last two terms of the paper .
Outcome:	the correct formula is now available on the web.

Copied to clipboard

Marina Sedinkina, Nikolas Breitkopf, Hinrich Schütze

GlotScript: A Resource and Tool for Low Resource Writing System Identification (2024.lrec-main)

Challenge:	In this paper, we automatically create sentiment dictionaries for predicting financial outcomes excess return and volatility.
Approach:	They propose to automatically adapt a domain-general dictionary to a financial domain and then manually adapt it to dictionaries for the finance domain.
Outcome:	The proposed dictionary outperforms the previous state of the art in predicting financial variables excess return and volatility.

Copied to clipboard

Amir Hossein Kargaran, François Yvon, Hinrich Schütze

A Graph Auto-encoder Model of Derivational Morphology (2020.acl-main)

Challenge:	GlotScript is an open resource and tool for low resource writing system identification.
Approach:	They propose to use GlotScript to automatically identify writing systems for low resource languages . they demonstrate that Glotscript can help cleaning multilingual corpora .
Outcome:	The proposed tool can help clean multilingual corpora and provide insights on coverage of low resource scripts and languages by each language model.

Copied to clipboard

Valentin Hofmann, Hinrich Schütze, Janet Pierrehumbert

Discrete and Soft Prompting for Multilingual Models (2021.emnlp-main)

Challenge:	Existing words that conform to morphological patterns of a language differ in how likely they are to be actually created by speakers.
Approach:	They propose to model the morphological well-formedness of derivatives by combining syntactic and semantic information with associative information from the mental lexicon.
Outcome:	The proposed model models the morphological well-formedness of derivatives in English .

Copied to clipboard

Mengjie Zhao, Hinrich Schütze

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2021.eacl-main)

Challenge:	In few-shot learning, discrete and soft prompting perform better than finetuning in multilingual cases.
Approach:	They show that discrete and soft prompting perform better than finetuning in crosslingual transfer and in-language training of multilingual natural language inference.
Outcome:	The proposed prompting model outperforms finetuning in crosslingual transfer and in-language training of multilingual natural language inference.

Copied to clipboard

Embedding Space Correlation as a Measure of Domain Similarity (2020.lrec-1)

Challenge:	Existing approaches to learning from examples are limited due to the vast number of languages, domains and tasks.
Approach:	They propose a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task.
Outcome:	The proposed approach outperforms supervised training and strong semi-supervised approaches in low-resource settings by a large margin.

Copied to clipboard

Anne Beyer, Göran Kauermann, Hinrich Schütze

Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings (P19-1)

Challenge:	Existing work on domain similarity using text-based features of corpus is limited by pre-trained word embeddings.
Approach:	They propose a measure of domain similarity based on dimension-wise correlations between embedding spaces . they find a threshold at which the measure indicates that two corpora come from the same domain .
Outcome:	The proposed measure can be used to determine which corpora are more similar to each other in a cross-domain sentiment detection task.

Copied to clipboard

Yadollah Yaghoobzadeh, Katharina Kann, T. J. Hazen, Eneko Agirre, Hinrich Schütze

The better your Syntax, the better your Semantics? Probing Pretrained Language Models for the English Comparative Correlative (2022.emnlp-main)

Challenge:	Empirical analysis of word embeddings of ambiguous words is limited by the small size of manually annotated resources and by the fact that word senses are treated as unrelated individual concepts.
Approach:	They present a large dataset based on manual Wikipedia annotations and word senses, where word sense from different words are related by semantic classes.
Outcome:	The proposed method can predict whether a word is single-sense or multi-sensor, if the sense is frequent, and it can predict rare senses.

Copied to clipboard

Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, Hinrich Schütze

Generating Datasets with Pretrained Language Models (2021.emnlp-main)

Challenge:	Construction Grammar posits constructions as the central building blocks of language . human-like performance of pretrained language models on many NLP tasks has been alleged .
Approach:	They propose to use construction grammar to posit constructions as the central building blocks of language . they conduct experiments with three pretrained language models to examine their ability to classify and understand English comparative correlative .
Outcome:	The proposed models are able to recognise the English comparative correlative (CC) but fail to use its meaning.

Copied to clipboard

TopicBERT for Energy Efficient Document Classification (2020.findings-emnlp)

Challenge:	Recent approaches to obtain high-quality sentence embeddings from pretrained language models require labeled data or finetuned on large set of labeles.
Approach:	They propose to use generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch and fine tune much smaller and more efficient models.
Outcome:	The proposed approach outperforms baselines on several semantic textual similarity datasets.

Copied to clipboard

Yatin Chaudhary, Pankaj Gupta, Khushbu Saxena, Vivek Kulkarni, Thomas Runkler, Hinrich Schütze

Information Retrieval

Challenge:	Prior work has noted that BERT’s computational cost grows quadratically with sequence length thus leading to longer training times, higher GPU memory constraints and carbon emissions.
Approach:	They propose to combine topic and language models to optimize the computational cost of fine-tuning for document classification by complementary learning.
Outcome:	The proposed model achieves a 1.4x speedup with 40% reduction in CO2 emission while retaining 99.9% performance over 5 datasets.

Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA (2020.findings-emnlp)

Copied to clipboard

Nina Poerner, Ulli Waltinger, Hinrich Schütze

Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts (N19-1)

Challenge:	Pretrained Language Models (PTLMs) are typically pretraining on target-domain text, which is expensive in terms of hardware, runtime and CO 2 emissions.
Approach:	They propose a faster, CPU-only domainadaptation method that trains Word2Vec on target-domain text and aligns the resulting word vectors with the wordpiece vectors of a general-domain PTLM.
Outcome:	The proposed method covers 60% of the BioBERT - BERT F1 delta, 5% of BioBERTS’s CO2 footprint and 2% of its cloud compute cost.

Copied to clipboard

A Multilingual BPE Embedding Space for Universal Sentiment Lexicon Induction (P19-1)

Challenge:	Mimicking has been proposed as a solution to learning high-quality embeddings for rare words because of sparse context information.
Approach:	They propose a method to reproduce embeddings of frequent words from their surface form and then use it to compute embedds for rare words.
Outcome:	The proposed model outperforms previous work on rare and medium-frequency words.

Copied to clipboard

Mengjie Zhao, Hinrich Schütze

Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging (N19-1)

Challenge:	Existing methods for sentiment lexicon induction are limited to low-resource languages.
Approach:	They propose a method for sentiment lexicon induction that is applicable to the entire range of typological diversity of the world's languages.
Outcome:	The proposed method is applicable to the entire range of typological diversity of the world's languages.

Copied to clipboard

Apostolos Kemos, Heike Adel, Hinrich Schütze

Robustness in NLP

Challenge:	Character-level models of tokens are effective at dealing with within-token noise and out-of-vocabulary words.
Approach:	They propose to eliminate the need for tokenizers by using a character-level semi-Markov conditional random field that uses neural networks for its character and segment representations.
Outcome:	The proposed model outperforms state-of-the-art part-of speech taggers on a noisy English dataset.

Combining Word Embeddings with Bilingual Orthography Embeddings for Bilingual Dictionary Induction (2020.coling-main)

Copied to clipboard

Silvia Severini, Viktor Hangya, Alexander Fraser, Hinrich Schütze

Quantifying the Contextualization of Word Representations with Semantic Class Probing (2020.findings-emnlp)

Challenge:	Bilingual dictionary induction (BDI) is a task of finding target language translations of source language words.
Approach:	They propose to use bilingual orthography Embeddings to enrich BWE-based BDI with transliteration information to make a decision on which information source is more reliable for a particular word pair.
Outcome:	The proposed system improves on English-Russian BDI and shows that it can be built with only weak bilingual signals and even without any bilingual signal.

Copied to clipboard

Mengjie Zhao, Philipp Dufter, Yadollah Yaghoobzadeh, Hinrich Schütze

Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages (2023.acl-long)

Challenge:	Pretrained language models are effective in solving NLP tasks, but there are still questions about how and why they work so well.
Approach:	They use BERT to quantify contextualization by studying the extent of inference . they show that top layer representations support highly accurate inference of semantic classes .
Outcome:	The proposed model is highly accurate, but weak in the lower layers . it is more task-specific after finetuning while lower layers are more transferable .

Copied to clipboard

Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, Hinrich Schütze

Fortification of Neural Morphological Segmentation Models for Polysynthetic Minimal-Resource Languages (N18-1)

Challenge:	Lack of LLMs supporting low-resource languages is a serious impediment to bringing NLP to all of the world.
Approach:	They create a model that scales LLMs horizontally and a corpus that covers 511 low-resource languages.
Outcome:	The proposed model improves on five diverse tasks across low- and high-resource languages.

Copied to clipboard

Katharina Kann, Jesus Manuel Mager Hois, Ivan Vladimir Meza-Ruiz, Hinrich Schütze

ParCourE: A Parallel Corpus Explorer for a Massively Multilingual Corpus (2021.acl-demo)

Challenge:	Morphological segmentation for polysynthetic languages is challenging because of limited training data.
Approach:	They propose two new multi-task training approaches that improve performance for Mexican polysynthetic languages . they also propose cross-lingual transfer as a third way to fortify their neural model .
Outcome:	The proposed models improve on Mexicanero, Nahuatl, Wixarika and Yorem Nokki . the proposed models reduce the amount of parameters by close to 75% .

Copied to clipboard

Ayyoob ImaniGooghari, Masoud Jalili Sabet, Philipp Dufter, Michael Cysou, Hinrich Schütze

Wine is not v i n. On the Compatibility of Tokenizations across Languages (2021.findings-emnlp)

Challenge:	7000 languages worldwide are spoken, but most research is focused on English . multilinguality is essential for multilingual research, and is a key component of the process.
Approach:	They propose a wordaligned parallel corpus that can be browsed using an online tool . they use the word alignment tools SimAlign and BabelNet to find the alignments .
Outcome:	The proposed tool can be set up for any parallel corpus and explores its quality and properties.

Copied to clipboard

Antonis Maronikolakis, Philipp Dufter, Hinrich Schütze

Multi-View Domain Adapted Sentence Embeddings for Low-Resource Unsupervised Duplicate Question Detection (D19-1)

Challenge:	Pretrained language models are used for performance and memory constraints, but there is little work that investigates the compatibility of tokenizations across languages.
Approach:	They propose a compatibility measure that reflects compatibility of tokenizations across languages.
Outcome:	The proposed measure prevents incompatible tokenizations, e.g., “wine” in English vs. “v i n” in French, which make it hard to learn good multilingual semantic representations.

Copied to clipboard

Nina Poerner, Hinrich Schütze

Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (2020.coling-main)

Challenge:	Stack Exchange has fewer than 160 user-labeled duplicates, and 25% have fewer.
Approach:	They propose a framework that combines sentence encoders with unlabeled data to solve the problem of duplicate question detection in Community Question Answering forums.
Outcome:	The proposed framework outperforms BM25, a single-view system and a supervised domain-adversarial DQD method on the CQADupStack corpus and on Stack Exchange forums.

Copied to clipboard

Timo Schick, Helmut Schmid, Hinrich Schütze

Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement (P18-1)

Challenge:	Existing approaches to few-shot text classification require domain expertise and an understanding of the language model's abilities to define the mapping between words and labels.
Approach:	They propose a method that converts textual inputs to cloze questions that contain some form of task description and processes them with a pretrained language model to map the predicted words to labels.
Outcome:	The proposed approach performs almost as well as hand-crafted label-to-word mappings for a number of tasks with small amounts of training data.

Copied to clipboard

Nina Poerner, Hinrich Schütze, Benjamin Roth

Explainability & Interpretability in NLP

Challenge:	a number of post hoc explanation methods for deep neural networks have been proposed . due to the complexity of the DNNs they explain, these methods are necessarily approximations and come with their own sources of error.
Approach:	They propose two evaluation paradigms that cover two important classes of NLP problems . they propose LIMSSE, LRP and DeepLIFT as the most effective explanation methods .
Outcome:	The proposed methods are most effective for explaining deep neural networks in NLP . the proposed methods can explain complex models without manual annotation .

SimAlign: High Quality Word Alignments Without Parallel Training Data Using Static and Contextualized Embeddings (2020.findings-emnlp)

Copied to clipboard

Masoud Jalili Sabet, Philipp Dufter, François Yvon, Hinrich Schütze

Towards a Broad Coverage Named Entity Resource: A Data-Efficient Approach for Many Diverse Languages (2022.lrec-1)

Challenge:	Word alignments are useful for statistical and neural machine translation (NMT) and cross-lingual annotation projection.
Approach:	They propose to leverage multilingual word embeddings for word alignment.
Outcome:	The proposed methods perform better for four languages and comparable for two languages than traditional statistical aligners even with abundant parallel data.

Copied to clipboard

Silvia Severini, Ayyoob ImaniGooghari, Philipp Dufter, Hinrich Schütze

Responsible & Trustworthy NLP

Challenge:	Existing methods to extract named entity datasets from parallel corpora require large monolingual corporata or word aligners that are unavailable or perform poorly for underresourced languages.
Approach:	They propose a method for creating a multilingual named entity resource from parallel corpora and apply it to the Parallel Bible Corpus, a corpus of more than 1000 languages.
Outcome:	The proposed method outperforms existing methods in two tasks.

ThaiLMCut: Unsupervised Pretraining for Thai Word Segmentation (2020.lrec-1)

Copied to clipboard

Suteera Seeha, Ivan Bilan, Liliana Mamani Sanchez, Johannes Huber, Michael Matuschek, Hinrich Schütze

Federated Continual Learning for Text Classification via Selective Inter-client Transfer (2022.findings-emnlp)

Challenge:	ThaiLMCut is a semi-supervised word segmentation model for word segmenting in Thai . it uses a bi-directional character language model to leverage useful linguistic knowledge from unlabeled data.
Approach:	They propose a semi-supervised approach to Thai word segmentation using a character language model.
Outcome:	The proposed approach outperforms state-of-the-art models on the benchmark InterBEST2009.

Copied to clipboard

Yatin Chaudhary, Pranav Rai, Matthias Schubert, Hinrich Schütze, Pankaj Gupta

BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance (2020.acl-main)

Challenge:	Continual Learning (CL) is a privacy-preserving machine learning technique that enables collaborative training of ML models by sharing model parameters across distributed clients.
Approach:	They propose a framework which selectively combines model parameters of foreign clients to maximize knowledge transfer while preserving privacy.
Outcome:	The proposed framework improves the performance of a text classification task using five datasets from diverse domains while preserving privacy.

Copied to clipboard

Two Methods for Domain Adaptation of Bilingual Tasks: Delightfully Simple and Broadly Applicable (P18-1)

Challenge:	Existing approaches to improve word embeddings for rare words are limited to uncontextualized word embeds.
Approach:	They propose a powerful architecture that can infer high-quality embeddings for rare words . they use the surface form and contexts of a word to interact in a deep architecture .
Outcome:	The proposed architecture can infer high-quality embeddings for rare words that are suitable as input representations for deep language models.

Copied to clipboard

Viktor Hangya, Fabienne Braune, Alexander Fraser, Hinrich Schütze

It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2021.naacl-main)

Challenge:	Previously, domain adaptation approaches to bilingual tasks were proposed . we show that simple adaptation process involving only unlabeled text is highly effective .
Approach:	They propose a method for domain adaptation of bilingual word embeddings using unlabeled data . they then tailor a semi-supervised classification method from computer vision to these tasks .
Outcome:	The proposed method improves on two bilingual tasks using unlabeled data.

Copied to clipboard

Neural Architectures for Fine-Grained Propaganda Detection in News (D19-50)

Challenge:	Pretraining ever-larger language models on massive corpora requires enormous amounts of compute.
Approach:	They propose to convert textual inputs into cloze questions that contain a task description . they also exploit unlabeled data to improve their performance .
Outcome:	The proposed model outperforms GPT-3 with PET/iPET with cloze questions and unlabeled data.

Copied to clipboard

Pankaj Gupta, Khushbu Saxena, Usama Yaseen, Thomas Runkler, Hinrich Schütze

Responsible & Trustworthy NLP

Challenge:	MIC-CIS is a fine grained propaganda detection system . previous work focused on document level, labeling articles as propaganda .
Approach:	They propose to use different neural architectures to jointly perform propaganda detection tasks . they also investigate different ensemble schemes such as majority-voting, relax-vote, etc.
Outcome:	The proposed system performs sentences and fragment level propaganda detection tasks.

True Few-Shot Learning with Prompts—A Real-World Perspective (2022.tacl-1)

Copied to clipboard

Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models (2021.eacl-main)

Challenge:	Recent work has cast doubt on the effectiveness of prompt-based approaches at few-shot learning in a “true” few- shot setting.
Approach:	They propose a method that combines textual instructions with example-based finetuning to give prompt-based learning a powerful method for few-shot text classification.
Outcome:	The proposed method performs well in a few-shot setting without a dev set and is able to handle multiple prompts.

Copied to clipboard

Nora Kassner, Philipp Dufter, Hinrich Schütze

Embedding Learning Through Multilingual Concept Induction (P18-1)

Challenge:	Recent work has shown that monolingual English language models can be used as knowledge bases.
Approach:	They use mBERT to query monolingual English language models with masked sentences to test their hypothesis.
Outcome:	The proposed model can be used to answer fill-in-the-blank questions in 53 languages.

Copied to clipboard

Philipp Dufter, Mengjie Zhao, Martin Schmitt, Alexander Fraser, Hinrich Schütze

PVGRU: Generating Diverse and Relevant Dialogue Responses via Pseudo-Variational Mechanism (2023.acl-long)

Challenge:	Existing methods for learning vector space representations of words are based on word-context information.
Approach:	They propose a method for estimating vector space representations of words by concept induction.
Outcome:	The proposed method performs better on crosslingual word similarity and sentiment analysis on a parallel corpus.

Copied to clipboard

Yongkang Liu, Shi Feng, Daling Wang, Yifei Zhang, Hinrich Schütze

Dialogue Systems & Conversational Agents

Challenge:	Existing generative models for dialogue use the last hidden state to summarize the history of the dialogue.
Approach:	They propose a Pseudo-Variational Gated Recurrent Unit (PVGRU) that summarises the accumulated distribution variations of subsequences and builds a model based on it.
Outcome:	The proposed model can improve diversity and relevance of responses on two benchmark datasets.

ECOLA: Enhancing Temporal Knowledge Embeddings with Contextualized Language Representations (2023.findings-acl)

Copied to clipboard

Zhen Han, Ruotong Liao, Jindong Gu, Yao Zhang, Zifeng Ding, Yujia Gu, Heinz Koeppl, Hinrich Schütze, Volker Tresp

MaiBaam: A Multi-Dialectal Bavarian Universal Dependency Treebank (2024.lrec-main)

Challenge:	Existing enhancement approaches cannot be applied to temporal knowledge graphs (tKGs) existing enhancement approaches assume knowledge embedding is time-independent, whereas entity embedded in tKG models evolves .
Approach:	They propose to use textual data to enhance temporal knowledge embedding by Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations (ECOLA) to evaluate ECOLA, they introduce three new datasets for training and evaluation.
Outcome:	The proposed model significantly improves Hits@1 on the link prediction task.

Copied to clipboard

Verena Blaschke, Barbara Kovačić, Siyao Peng, Hinrich Schütze, Barbara Plank

BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief (2021.emnlp-main)

Challenge:	Despite the success of the Universal Dependencies (UD) project, there is still a lack of diversity within high-resource languages and their closely related non-standard languages and dialects.
Approach:	They propose to annotate Bavarian with part-of-speech and syntactic dependency information manually in UD and to highlight morphosyntactical differences between the closely related languages.
Outcome:	The proposed treebank covers multiple genres including wiki, fiction, grammar examples, social, non-fiction and Bavarian.

Copied to clipboard

Nora Kassner, Oyvind Tafjord, Hinrich Schütze, Peter Clark

DagoBERT: Generating Derivational Morphology with a Pretrained Language Model (2020.emnlp-main)

Challenge:	Pretrained language models can produce inconsistent answers when probed, even after specialized training.
Approach:	They propose to embed a pretrained language model in a broader system that includes an evolving, symbolic memory of beliefs that records but may modify the raw PTLM answers.
Outcome:	The proposed architecture improves belief consistency in the overall system by revising beliefs that clash with others and generating queries using known beliefs as context.

Copied to clipboard

Valentin Hofmann, Janet Pierrehumbert, Hinrich Schütze

Increasing Learning Efficiency of Self-Attention Networks through Direct Position Interactions, Learnable Temperature, and Convoluted Attention (2020.coling-main)

Challenge:	Pretrained language models (PLMs) generate derivationally complex words, but it is unclear what they learn about other aspects of language.
Approach:	They propose to use BERT to examine its derivational capabilities in different settings, from unmodified pretrained models to full finetuning.
Outcome:	The proposed model outperforms the state-of-the-art in derivation generation.

Copied to clipboard

Philipp Dufter, Martin Schmitt, Hinrich Schütze

Neural Transductive Learning and Beyond: Morphological Generation in the Minimal-Resource Setting (D18-1)

Challenge:	SANs are an integral part of successful neural networks such as Transformer . training SAN on a task or pretraining them on language modeling requires large amounts of data and compute resources.
Approach:	They propose to modify SANs to enable faster learning, i.e., higher accuracies after fewer update steps.
Outcome:	The proposed modifications enable faster learning, i.e., higher accuracies after fewer update steps.

Copied to clipboard

Katharina Kann, Hinrich Schütze

Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020.emnlp-main)

Challenge:	Existing lexicons have limited coverage for learning morphological inflection patterns from labeled data.
Approach:	They propose two new methods to solve paradigm completion, the morphological task of generating missing forms, given a partial paradigm.
Outcome:	The proposed methods outperform the previous state-of-the-art by 9.71% absolute accuracy on a 52-language benchmark dataset.

Copied to clipboard

Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, Hinrich Schütze

Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons (2024.lrec-main)

Challenge:	Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that our binary masked language models encode information necessary for solving downstream tasks.
Approach:	They propose an efficient method of utilizing pretrained language models where selective binary masks are learned instead of finetuning.
Outcome:	Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that the proposed method yields comparable performance to finetuning, but has a much smaller memory footprint when multiple tasks need to be solved.

Copied to clipboard

Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze, David R. Mortensen, Lori Levin

Identifying Elements Essential for BERT’s Multilinguality (2020.emnlp-main)

Challenge:	In this paper, we examine the ability of large language models (LLMs) to identify different meanings in sentences that are superficially similar.
Approach:	They propose a challenge dataset for NLP with large lexical overlap which minimises the possibility of models discerning entailment solely based on token distinctions.
Outcome:	The proposed model fails to distinguish between constructions with three classes of adjectives which cannot be distinguished by surface features.

Copied to clipboard

Philipp Dufter, Hinrich Schütze

Hengam: An Adversarially Trained Transformer for Persian Temporal Tagging (2022.aacl-main)

Challenge:	Multilingual BERT (mBERT) does not use any crosslingual signal during training.
Approach:	They propose a multilingual pretraining setup that modifies the masking strategy using VecMap to allow for fast experimentation.
Outcome:	The proposed setup with pretrained models with three languages shows that it works well.

Copied to clipboard

Sajad Mirzababaei, Amir Hossein Kargaran, Hinrich Schütze, Ehsaneddin Asgari