Comparing and Developing Tools to Measure the Readability of Domain-Specific Texts (D19-1)
Copied to clipboard
Elissa Redmiles, Lisa Maszkiewicz, Emily Hwang, Dhruv Kuchhal, Everest Liu, Miraida Morales, Denis Peskov, Sudha Rao, Rock Stevens, Kristina Gligorić, Sean Kross, Michelle Mazurek, Hal Daumé III
| Challenge: | Despite this, we lack a thorough understanding of how to validly measure readability at scale, especially for domain-specific texts. |
| Approach: | They present a comparison of the validity of well-known readability measures and introduce a novel approach to measure readability at scale. |
| Outcome: | The proposed approach addresses shortcomings of existing measures. |
Similar Papers
Trends, Limitations and Open Challenges in Automatic Readability Assessment Research (2022.lrec-1)
Copied to clipboard
| Challenge: | Readability assessment is the task of evaluating the reading difficulty of a given piece of text. |
| Approach: | They examine the common approaches used for automatic readability assessment and identify their shortcomings and some challenges for the future. |
| Outcome: | The proposed models are compared with existing models and are based on existing ones. |
Evaluating the Evaluators: Are readability metrics good measures of readability? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Plain language summarization (PLS) aims to distill complex documents into accessible summaries for non-expert audiences. |
| Approach: | They conduct a thorough survey of literature on plain language summarization (PLS) and find that traditional readability metrics are not compared to human judgments. |
| Outcome: | The proposed language models better capture deeper measures of readability, with the best-performing model achieving a Pearson correlation of 0.56 with human judgments. |
Metrics for What, Metrics for Whom: Assessing Actionability of Bias Evaluation Metrics in NLP (2024.emnlp-main)
Copied to clipboard
| Challenge: | a measure’s intended use and reliability assessment are often unclear or entirely absent from the literature examining bias measures in natural language processing. |
| Approach: | They propose a set of desiderata to assess the degree to which a measure’s results enable informed action and a review of 146 papers proposing bias measures in NLP. |
| Outcome: | The proposed desiderata are based on 146 papers proposing bias measures in natural language processing (NLP) . they show that key elements of actionability, including a measure’s intended use and reliability assessment, are often unclear or entirely absent. |
Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks, but statistical bias in benchmark data and probing studies has recently called into question their true capabilities. |
| Approach: | They propose to evaluate systems through a measure of prediction coherence by using two existing language understanding benchmarks with different properties to demonstrate its versatility. |
| Outcome: | The proposed evaluation framework is quick, effective, and versatile to provide insight into the coherence of machines’ predictions. |
Interpretable Text Embeddings and Text Similarity Explanation: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text embeddings are a fundamental component in many NLP tasks, but their interpretation and explanation remain challenging. |
| Approach: | They propose a framework for interpretable text embeddings and text similarity explanation . they characterize the main ideas, approaches, and trade-offs and discuss lessons learned . |
| Outcome: | The proposed methods are compared with existing models and compare them with existing ones. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features (2021.emnlp-main)
Copied to clipboard
| Challenge: | ML models with handcrafted features are linguistically explainable, expandable, and competent against the modern neural models. |
| Approach: | They propose to combine traditional ML models with ML transformers to improve readability assessment by 99% accuracy. |
| Outcome: | The proposed model achieves state-of-the-art (SOTA) accuracy on popular datasets. |
Uncertainty in Language Models: Assessment through Rank-Calibration (2024.emnlp-main)
Copied to clipboard
Xinmeng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, Edgar Dobriban
| Challenge: | Language Models (LMs) have shown promising performance in natural language generation . however, it is crucial to correctly quantify their level of uncertainty in responding to inputs. |
| Approach: | They propose a framework to quantify uncertainty and confidence for Large Language Models . they use a Rank-calibration framework to measure uncertainty and confident responses . |
| Outcome: | The proposed framework assesses uncertainty and confidence measures for LMs. |
Semantic Accuracy in Natural Language Generation: A Thesis Proposal (2023.acl-srw)
Copied to clipboard
| Challenge: | Using large pre-trained language models, it is essential to research their reliability . if a human does not know the answer to a question, the socially acceptable behavior is to say 'I do not know' failing to fulfill this expectation can lead to distrust, or spread of misinformation. |
| Approach: | They propose a method for evaluating semantic accuracy and a benchmark for NLG metrics. |
| Outcome: | The proposed method evaluates semantic accuracy and provides a benchmark for NLG metrics. |