Papers by Tomasz Limisiewicz
Universal Dependencies According to BERT: Both More Specific and More General (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that individual BERT heads encode particular dependency relation types, but they do not match one-to-one. |
| Approach: | They propose a method for relation identification and syntactic tree construction that can be applied with minimal supervision and generalizes well across languages. |
| Outcome: | The proposed method produces significantly more consistent dependency trees than previous work and can be applied with only a minimal amount of supervision and generalizes well across languages. |
Beyond Literal Token Overlap: Token Alignability for Multilinguality (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that token overlap is a strong predictor of multilinguality and cross-lingual knowledge transfer between languages with different scripts. |
| Approach: | They propose a subword token alignability metric to understand the impact and quality of multilingual tokenisation. |
| Outcome: | The proposed metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low. |
A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank (2022.naacl-main)
Copied to clipboard
| Challenge: | Pretraining languages improve cross-lingual transfer for BERT-based models . Interestingly, PLMs exhibit zero-shot cross-linguistic abilities on downstream examples in languages seen only during pretraining. |
| Approach: | They develop a quadratic time complexity method to estimate pretraining languages' relations between linguistic features and two downstream tasks. |
| Outcome: | The proposed method is effective on a diverse set of languages spanning different linguistic features and two downstream tasks. |
Introducing Orthogonal Constraint in Structural Probes (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on interpreting pre-trained models' representations and analyzing their structures. |
| Approach: | They propose a new type of structural probing where a linear projection is decomposed into two types. |
| Outcome: | The proposed method is tested on two novel tasks and shows that lexical and syntactic information is separated in the representations. |
Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models (2024.emnlp-main)
Copied to clipboard
Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah Smith, Luke Zettlemoyer
| Challenge: | Multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. |
| Approach: | They propose Cross-lingual Expert Language Models (X-ELM) which mitigates inter-language competition by independently training language models on subsets of the multilingual corpus. |
| Outcome: | The proposed model outperforms jointly trained multilingual models across all 16 considered languages and transfer the gains to downstream tasks. |
Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | Multilingual language models perform surprisingly well in a variety of NLP tasks for diverse languages. |
| Approach: | They propose to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. |
| Outcome: | The proposed criteria show that the overlap of vocabulary across languages can be detrimental to certain downstream tasks. |
Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on whether multilingual embeddings can be aligned in a shared space across languages are lacking. |
| Approach: | They propose to learn a projection based on monolingual annotated datasets and evaluate syntactic and lexical information encoded in a shared cross-lingual embedding space. |
| Outcome: | The proposed model can be used to learn representations for languages with low resources. |