Papers by Philipp Dufter
Monolingual and Multilingual Reduction of Gender Bias in Contextualized Representations (2020.coling-main)
Copied to clipboard
| Challenge: | Prior work identifies a linear gender subspace and removes gender information by eliminating the subspace. |
| Approach: | They propose to use DensRay to obtain interpretable dense subspaces by applying it to attention heads and layers of BERT. |
| Outcome: | The proposed method performs on-par with prior approaches, but is more robust and preserves language model performance better. |
Graph Algorithms for Multiparallel Word Alignment (2021.emnlp-main)
Copied to clipboard
Ayyoob ImaniGooghari, Masoud Jalili Sabet, Lutfi Kerem Senel, Philipp Dufter, François Yvon, Hinrich Schütze
| Challenge: | Word alignments are useful for typological research and can be used in machine translation systems. |
| Approach: | They propose to exploit the multiparallelity of parallel corpora by representing bilingual alignments as a graph and then predicting additional edges. |
| Outcome: | The proposed algorithm improves the accuracy of bilingual alignments by 28% over baseline algorithms. |
Static Embeddings as Efficient Knowledge Bases? (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent research investigates factual knowledge stored in large pretrained language models . masked sentences such as “Paris is the capital of [MASK]” are used as probes . |
| Approach: | They use masked sentences to test whether a language model can capture factual knowledge . they show that static embeddings perform better than PLMs when restricted to a candidate set . |
| Outcome: | The results show that static embeddings perform better than PLMs when restricted to a candidate set . |
Quantifying the Contextualization of Word Representations with Semantic Class Probing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models are effective in solving NLP tasks, but there are still questions about how and why they work so well. |
| Approach: | They use BERT to quantify contextualization by studying the extent of inference . they show that top layer representations support highly accurate inference of semantic classes . |
| Outcome: | The proposed model is highly accurate, but weak in the lower layers . it is more task-specific after finetuning while lower layers are more transferable . |
ParCourE: A Parallel Corpus Explorer for a Massively Multilingual Corpus (2021.acl-demo)
Copied to clipboard
| Challenge: | 7000 languages worldwide are spoken, but most research is focused on English . multilinguality is essential for multilingual research, and is a key component of the process. |
| Approach: | They propose a wordaligned parallel corpus that can be browsed using an online tool . they use the word alignment tools SimAlign and BabelNet to find the alignments . |
| Outcome: | The proposed tool can be set up for any parallel corpus and explores its quality and properties. |
Wine is not v i n. On the Compatibility of Tokenizations across Languages (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models are used for performance and memory constraints, but there is little work that investigates the compatibility of tokenizations across languages. |
| Approach: | They propose a compatibility measure that reflects compatibility of tokenizations across languages. |
| Outcome: | The proposed measure prevents incompatible tokenizations, e.g., “wine” in English vs. “v i n” in French, which make it hard to learn good multilingual semantic representations. |
An Information-Theoretic Approach and Dataset for Probing Gender Stereotypes in Multilingual Masked Language Models (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) have been shown to encapsulate social biases, including those relating to gender and race. |
| Approach: | They propose a new bias measure based on Jensen–Shannon divergence that retains more information from the model output probabilities than other previously proposed bias measures. |
| Outcome: | The proposed measure outperforms CrowS-Pairs and other similar measures for non-English datasets. |
SimAlign: High Quality Word Alignments Without Parallel Training Data Using Static and Contextualized Embeddings (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Word alignments are useful for statistical and neural machine translation (NMT) and cross-lingual annotation projection. |
| Approach: | They propose to leverage multilingual word embeddings for word alignment. |
| Outcome: | The proposed methods perform better for four languages and comparable for two languages than traditional statistical aligners even with abundant parallel data. |
Towards a Broad Coverage Named Entity Resource: A Data-Efficient Approach for Many Diverse Languages (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods to extract named entity datasets from parallel corpora require large monolingual corporata or word aligners that are unavailable or perform poorly for underresourced languages. |
| Approach: | They propose a method for creating a multilingual named entity resource from parallel corpora and apply it to the Parallel Bible Corpus, a corpus of more than 1000 languages. |
| Outcome: | The proposed method outperforms existing methods in two tasks. |
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent work has shown that monolingual English language models can be used as knowledge bases. |
| Approach: | They use mBERT to query monolingual English language models with masked sentences to test their hypothesis. |
| Outcome: | The proposed model can be used to answer fill-in-the-blank questions in 53 languages. |
Embedding Learning Through Multilingual Concept Induction (P18-1)
Copied to clipboard
| Challenge: | Existing methods for learning vector space representations of words are based on word-context information. |
| Approach: | They propose a method for estimating vector space representations of words by concept induction. |
| Outcome: | The proposed method performs better on crosslingual word similarity and sentiment analysis on a parallel corpus. |
Increasing Learning Efficiency of Self-Attention Networks through Direct Position Interactions, Learnable Temperature, and Convoluted Attention (2020.coling-main)
Copied to clipboard
| Challenge: | SANs are an integral part of successful neural networks such as Transformer . training SAN on a task or pretraining them on language modeling requires large amounts of data and compute resources. |
| Approach: | They propose to modify SANs to enable faster learning, i.e., higher accuracies after fewer update steps. |
| Outcome: | The proposed modifications enable faster learning, i.e., higher accuracies after fewer update steps. |
Identifying Elements Essential for BERT’s Multilinguality (2020.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) does not use any crosslingual signal during training. |
| Approach: | They propose a multilingual pretraining setup that modifies the masking strategy using VecMap to allow for fast experimentation. |
| Outcome: | The proposed setup with pretrained models with three languages shows that it works well. |
Analytical Methods for Interpretable Ultradense Word Embeddings (D19-1)
Copied to clipboard
| Challenge: | Word embeddings lack interpretability, but rotation of word spaces can help . e.g., lexicon induction, gender bias can be removed by removing interpretable dimensions . |
| Approach: | They propose three methods for making word embeddings interpretable by rotation . they use Densifier, linear SVMs and DensRay to compute word spaces in closed form . |
| Outcome: | The proposed method can be computed in closed form and is more robust than Densifier. |