Unsupervised Evaluation Metrics and Learning Criteria for Non-Parallel Textual Transfer (D19-56)
Copied to clipboard
| Challenge: | Existing methods for textual transfer with no parallel corpora are insufficient to evaluate textual paraphrases with modified attributes or properties. |
| Approach: | They propose to add a metric for post-transfer classification accuracy and a method to combine them into a single overall score. |
| Outcome: | The proposed metrics correlate well with human judgments, at both the sentence-level and system-level. |
Similar Papers
Towards Actual (Not Operational) Textual Style Transfer Auto-Evaluation (D19-55)
Copied to clipboard
| Challenge: | elucidates the dangerous current state of style transfer auto-evaluation research. |
| Approach: | They propose ways to aggregate the three metrics into one evaluator. |
| Outcome: | The proposed method could be used to aggregate the three metrics into one evaluator. |
A large-scale computational study of content preservation measures for text style transfer and paraphrase generation (2022.acl-srw)
Copied to clipboard
| Challenge: | Text style transfer and paraphrases generation are growing areas of NLP . many researchers still use BLEU-like measures to evaluate content preservation . |
| Approach: | They compare 57 different measures based on different principles on 19 annotated datasets . they find that measures relying on cross-encoder models outperform alternative approaches . |
| Outcome: | The proposed methods outperform traditional methods on 19 datasets. |
Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer (2021.emnlp-main)
Copied to clipboard
| Challenge: | a lack of standardized and reliable methods for automatic evaluation hinders ST . prior work has employed as many as nine different automatic systems to rate formality alone . |
| Approach: | They evaluate automatic metrics on the oft-researched task of formality style transfer . they outline best practices for automatic evaluation in (formality) style transfer and identify models that correlate well with human judgments. |
| Outcome: | The proposed models correlate well with human judgments and are robust across languages. |
UScore: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | supervised evaluation metrics are not available for machine translation, despite their wide dissemination. |
| Approach: | They develop fully unsupervised evaluation metrics that leverage parallel data and evaluation metric induction. |
| Outcome: | The proposed metrics beat supervised competitors on 4 out of 5 evaluation datasets. |
Rethinking Sentiment Style Transfer (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods for text style transfer are unsatisfactory. |
| Approach: | They propose to use a graph-based method to extract attribute content from sentences . they propose an efficient regularization to leverage attribute-dependent content as guiding signals. |
| Outcome: | The proposed method is based on a YELP and IMDB dataset and it is able to detect errors in the human evaluation. |
IMaT: Unsupervised Text Attribute Transfer via Iterative Matching and Translation (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to rewrite sentences with certain attributes are difficult and often result in poor content-preservation and ungrammaticality. |
| Approach: | They propose a method that uses a sequence-to-sequence model to learn attribute transfer . existing approaches try to explicitly disentangle content and attribute information . |
| Outcome: | The proposed method outperforms complex state-of-the-art systems by a large margin in sentiment modification and formality transfer tasks. |
Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics? (2025.naacl-srw)
Copied to clipboard
| Challenge: | Text style transfer (TST) is a multidimensional task requiring the assessment of style transfer accuracy, content preservation, and naturalness. |
| Approach: | They propose to use text style transfer metrics to evaluate outputs of text editors . they also investigate the potential of large language models as tools for TST evaluation . |
| Outcome: | The proposed methods provide better insights than existing metrics, the authors show . their meta-evaluation through correlation with hu-man judgments shows they are effective . |
On Learning Text Style Transfer with Direct Rewards (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for text style transfer lack parallel corpora, which makes it impossible to train supervised models. |
| Approach: | They propose to use semantic similarity metrics to explicitly assess the preservation of content between system outputs and inputs. |
| Outcome: | The proposed methods provide significant gains in automatic and human evaluation over strong baselines. |
On the Evaluation Metrics for Paraphrase Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for paraphrase generation are not designed for the task, but adopted from other evaluation tasks. |
| Approach: | They propose a new evaluation metric for paraphrase generation that uses reference-based and reference-free metrics. |
| Outcome: | The proposed evaluation metric outperforms existing metrics and is more reliable than reference-based metrics. |
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for style transfer warp the input’s meaning through attribute transfer, which changes semantic properties such as sentiment. |
| Approach: | They propose a method for fine-tuning pretrained language models on automatically generated paraphrase data to improve the efficiency of style transfer. |
| Outcome: | The proposed method outperforms state-of-the-art style transfer systems on human and automatic evaluations and proposes fixed variants. |