Papers by Daniel Rotem
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models use the attention mechanism to contextualize input inputs . but, we find that it is not as important as thought for pretrained models . |
| Approach: | They propose a probing method that replaces input-dependent attention matrices with constant ones. |
| Outcome: | The proposed method improves performance of pretrained language models without input-dependent attention. |
Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained Transformer-based language models such as BERT, DeBERTa, and GPT3 have become the go-to tool in NLP. |
| Approach: | They propose an Early-Exit fine-tuning method that assigns each classifier its own set of unique model weights, not updated by other classifiers. |
| Outcome: | The proposed method outperforms Early-Exit and Multi-Model at fast speeds while maintaining comparable scores to Early- Exit at slow speeds. |
On the Limitations of Reference-Free Evaluations of Generated Text (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has shown that evaluation metrics which accurately estimate the quality of generated text are limited in their ability to evaluate generated text. |
| Approach: | They argue that reference-free metrics are limited in their ability to evaluate generated text . they recommend that they be used as diagnostic tools for analyzing and understanding model behavior . |
| Outcome: | The proposed evaluation metrics are limited in their ability to evaluate generated text . they can be optimized at test time, can be biased against models with similar outputs . |
A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for summarization evaluations that approximate human judgments are lacking for accuracy and reliability. |
| Approach: | They propose methods for calculating confidence intervals and running hypothesis tests for correlations using bootstrapping and permutation. |
| Outcome: | The proposed methods show that the confidence intervals are wide, demonstrating high uncertainty in the reliability of automatic metrics. |
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing definitions of system-level correlations are inconsistent with how they are used to evaluate systems. |
| Approach: | They propose to calculate correlations only on pairs of systems separated by small differences in automatic scores . they propose to use the full test set instead of the subset of summaries judged by humans . |
| Outcome: | The proposed changes improve the accuracy of the estimated correlations on pairs of systems separated by small differences in automatic scores. |