Papers by Matan Eyal
Multilingual Sequence-to-Sequence Models for Hebrew NLP (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work on pretrained language models for Hebrew is under-parameterized and under-trained . previous work on pretraining Hebrew LMs focused on encoder-only architectures . |
| Approach: | They propose to use sequence-to-sequence generative architectures to train large LMs in morphologically rich languages such as Hebrew. |
| Outcome: | The proposed model improves on all existing Hebrew NLP benchmarks. |
Large Scale Substitution-based Word Sense Induction (2022.acl-long)
Copied to clipboard
| Challenge: | Word forms are ambiguous, and derive meaning from the context in which they appear . word sense induction can be performed over a corpus-derived sense inventory . |
| Approach: | They propose a word-sense induction method based on pre-trained masked language models . they train a static word embeddings algorithm on the sense-tagged corpus . |
| Outcome: | The proposed method outperforms existing senseful embeddings methods on Wikipedia and on an outlier detection dataset. |
Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications? (2024.naacl-short)
Copied to clipboard
Yotam Intrator, Matan Halfon, Roman Goldenberg, Reut Tsarfaty, Matan Eyal, Ehud Rivlin, Yossi Matias, Natalia Aizenberg
| Challenge: | Existing studies have focused on pre-translation, but there is still need for it . authors say that it is not universally necessary to translate large language models . |
| Approach: | They re-evaluate the need for pre-translation in the context of PaLM2 models . authors found that PaLM2-L consistently outperforms pre-translated in 94 out of 108 languages . |
| Outcome: | The proposed model outperforms pre-translation in 94 out of 108 languages and 6 benchmarks . authors argue that pre-translated inputs can be used to improve performance . |
Question Answering as an Automatic Evaluation Metric for News Article Summarization (N19-1)
Copied to clipboard
| Challenge: | Recent work on summarization and headline generation focuses on maximizing ROUGE scores. |
| Approach: | They propose an extrinsic evaluation metric that maximizes ROUGE scores for automatic summarization and headline generation. |
| Outcome: | The proposed model maximizes ROUGE scores while increasing competitive results. |
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-training Large Language Models (LLMs) on textual corpora embeds substantial factual knowledge in their parameters, which is essential for excelling in various downstream applications. |
| Approach: | They propose to use supervised fine-tuning to align large language models to new factual information that is not acquired through pre-training. |
| Outcome: | The proposed model is trained to generate facts that are not grounded in pre-existing knowledge, but hallucinates when examples with new knowledge are learned. |
Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs (2026.acl-long)
Copied to clipboard
Guy Mor-Lan, Omer Goldman, Matan Eyal, Adi Mayrav Gilady, Sivan Eiger, Idan Szpektor, Avinatan Hassidim, Yossi Matias, Reut Tsarfaty
| Challenge: | Multilingual large language models have minimized the fluency gap between languages, but they are exposed to the risk of biases as knowledge and norms may propagate across languages. |
| Approach: | They propose a test set with 2,156 questions in 12 languages to quantify models' biases . they show a global bias towards answers relevant to the US-locale . |
| Outcome: | The proposed model can answer locale-ambiguous questions in 12 languages. |
The Hidden Space of Transformer Language Adapters (2024.acl-long)
Copied to clipboard
| Challenge: | Adapters are small modules trained on top of a frozen language model to adapt predictions to new target languages. |
| Approach: | They propose to train transformer language adapters on top of a frozen model to adapt predictions to new target languages. |
| Outcome: | The transformer language adapters are trained on top of a frozen model to adapt predictions to new target languages. |
Bootstrapping Relation Extractors using Syntactic Search by Examples (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for supervised relation extraction still require a large quantity of training data. |
| Approach: | They propose a process for bootstrapping training datasets which can be performed quickly by non-NLP-experts. |
| Outcome: | The proposed method outperforms models trained on manual and distant data augmentation techniques and the search-based approach with the NLG method. |
Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance (2024.findings-acl)
Copied to clipboard
| Challenge: | Despite being the cornerstone of BPE, the importance of compression in the tokenization process is still unclear. |
| Approach: | They argue for the theoretical importance of compression in the tokenization process . they also demonstrate the empirical importance of compressing tokenizers for downstream success of pre-trained language models. |
| Outcome: | The proposed method can be viewed as 0-gram language modeling where equal probability is assigned to all tokens. |
Multilingual Instruction Tuning With Just a Pinch of Multilinguality (2024.findings-acl)
Copied to clipboard
| Challenge: | Using multilingual instruction tuning, large language models can be used to follow instructions in multiple languages . a multilingual model can be tuned on a wide range of languages, yet most datasets are limited to English . |
| Approach: | They investigate how multilinguality during instruction tuning affects instruction-following across languages . they find that only 40 multilingual examples improve multilingual instruction- follow . |
| Outcome: | The results show that multilingual models perform better on multilingual mixtures compared to monolingual models . the results suggest that building multilingual instruction-tuned models can be done with only 2-4 languages . |