Papers by Sara Rajaee
How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are ineffective, despite their potential, pre-training models suffer from important weaknesses. |
| Approach: | They analyze the extent to which the isotropy of the embedding space changes after fine-tuning. |
| Outcome: | The proposed model improves the isotropy of embedding space after fine-tuning . the model can encode linguistic properties, but lacks the social bias needed to improve it . |
Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies have focused on improving reasoning ability in English models, with multilingual models receiving comparatively little attention. |
| Approach: | They propose a framework that ranks candidate reasoning traces across languages rather than within a single language. |
| Outcome: | The proposed framework improves accuracy by up to 10 points in English compared to using reward modeling within a single language. |
Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for debiasing are ineffective in addressing the reverse word-overlap bias. |
| Approach: | They propose to investigate the reverse word-overlap bias in NLI models . they find that existing debiasing methods are generally ineffective . |
| Outcome: | The proposed model is biased towards the non-entailment label on instances with low overlap . the proposed model does not have minority examples, the authors show . |
An Isotropy Analysis in the Multilingual BERT Embedding Space (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have explored the advantages of multilingual pre-trained models in capturing shared linguistic knowledge. |
| Approach: | They investigate the anisotropic embedding space and outlier dimensions of the multilingual BERT model for two known issues of the monolingual models. |
| Outcome: | The proposed model has no outlier dimension and has highly anisotropic space . the results show that increasing the isotropy of multilingual space can improve its representation power and performance, similar to what had been observed for monolingual CWRs on semantic similarity tasks. |
Local Look-Ahead Guidance via Verifier-in-the-Loop for Automated Theorem Proving (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent methods for AI reasoning require applying variants of reinforcement learning (RL) on rolled out trajectories, even for step-wise rewards, or large quantities of human-annotated trajectory data. |
| Approach: | They propose a verifier-in-the-loop design that uses an automated verifier to give intermediate feedback at each step of the reasoning process. |
| Outcome: | The proposed model improves on the Automatic Theorem Proving task using Lean as the verifier. |
An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning is a computationally expensive process . introducing new parameters to an already-large model can be considered a drawback. |
| Approach: | They investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task. |
| Outcome: | The proposed methods show that each transformer module is a winning ticket . they show that with only 0.003% updateable parameters, they can show acceptable performance on target tasks. |
A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to address the representation degeneration problem in contextual embedding spaces require a learning process to retrain models with additional objectives. |
| Approach: | They propose a local cluster-based method to address the representation degeneration problem in contextual embedding spaces by removing local dominant directions from verb representations. |
| Outcome: | The proposed method improves CWRs performance on semantic tasks by removing dominant directions of verb representations. |
Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in training multilingual models on large datasets have shown promising results in knowledge transfer across languages. |
| Approach: | They challenge the assumption that high zero-shot performance reflects high cross-lingual ability by introducing more challenging setups involving instances with multiple languages. |
| Outcome: | The proposed model can achieve high performance on multilingual benchmarks and on low-resource languages. |
On the Importance of Data Size in Probing Fine-tuned Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. |
| Approach: | They propose to investigate the reasons behind the effectiveness of fine-tuning by examining the impact of data size on the extent of encoded linguistic knowledge. |
| Outcome: | The proposed probes show that the size of the training data affects the recoverability of the changes made to the model’s linguistic knowledge. |