New Evaluation Methodology for Qualitatively Comparing Classification Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Text Classification is one of the most common tasks in Natural Language Processing. |
| Approach: | They propose a method for performing qualitative assessment over multiple classification models using a fine-tuned BERT and Logistic Regression evaluation methodology. |
| Outcome: | The proposed evaluation methodology outperforms the baseline model in linguistic clustering and Sentiment Analysis. |
Similar Papers
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve text classification performance of pre-trained models have been used to improve their performance. |
| Approach: | They propose a method for improving BERT's performance by using a label embedding technique while keeping almost the same computational cost. |
| Outcome: | The proposed method improves BERT's performance on six text classification benchmark datasets while keeping almost the same computational cost. |
Multi-source Multi-domain Sentiment Analysis with BERT-based Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Sentiment analysis is a widely studied task in natural language processing. |
| Approach: | They propose to improve BERT-based models for sentiment analysis on italian corpora and evaluate their performance on the basis of eight corpors. |
| Outcome: | The proposed model is evaluated over eight sentiment analysis corpora from different domains and sources on the prediction of positive, negative and neutral classes. |
Linear Classifier: An Often-Forgotten Baseline for Text Classification (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models such as BERT are popular solutions for text classification. |
| Approach: | They argue that large-scale pre-trained language models such as BERT are popular solutions for text classification . authors argue that running a simple baseline like linear classifiers on bag-of-words features is important for text classification . |
| Outcome: | The proposed approach may only sometimes get satisfactory results for some problems. |
Casting the Same Sentiment Classification Problem (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Identifying the stance of an argument towards a topic is a fundamental problem in computational argumentation. |
| Approach: | They propose a task where text users are asked to determine if they have the same sentiment . they aim to enable a more topic-agnostic sentiment classification by using Yelp data . |
| Outcome: | The proposed task achieves an accuracy above 83% for category subsets across topics and 89% on average. |
Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself. |
| Approach: | This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods. |
| Outcome: | The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods . |
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021.emnlp-main)
Copied to clipboard
| Challenge: | Evaluation metrics are a key ingredient for progress of text generation systems . a class of novel evaluation metrics based on BERT and its variants has been explored . |
| Approach: | They propose to disentangle BERT-based evaluation metrics along linguistic factors . they show they are sensitive to lexical overlap, just like BLEU and ROUGE . |
| Outcome: | The proposed metrics capture all aspects but are sensitive to lexical overlap, just like BLEU and ROUGE, the authors show . |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework (2025.findings-acl)
Copied to clipboard
Kaishuai Xu, Tiezheng Yu, Yi Cheng, Wenjun Hou, Liangyou Li, Xin Jiang, Lifeng Shang, Qun Liu, Wenjie Li
| Challenge: | Existing methods for fine-tuning open-source LLMs are limited to text-based analysis under predefined general criteria. |
| Approach: | They propose a framework that fine-tunes LLMs to replicate the evaluation explanations and judgments of proprietary models. |
| Outcome: | The proposed evaluation framework outperforms existing fine-tuned evaluation methods in effectiveness and robustness. |
Comparing Text Representations: A Theory-Driven Approach (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in NLP have been made by learning representations that transform complex tasks into simple classification tasks. |
| Approach: | They propose a method to evaluate the compatibility between representations and tasks by fitting text features to specific characteristics of text datasets. |
| Outcome: | The proposed model provides a calibrated, quantitative measure of the difficulty of a classification-based NLP task. |