Rodrigo Wilkens, David Alfter, Xiaoou Wang, Alice Pintard, Anaïs Tack, Kevin P. Yancey, Thomas François
| Challenge: | a large number of readability predictor variables are used to predict reading difficulty of texts . the most important predictors for native texts are lexical diversity, dependency counts and text coherence . |
| Approach: | They propose a readability toolkit based on aggregation of readability predictor variables . they show which features are most predictive on two different corpora . |
| Outcome: | The proposed toolkit improves performance over standard feature-based readability prediction. |
Similar Papers
Comparing and Developing Tools to Measure the Readability of Domain-Specific Texts (D19-1)
Copied to clipboard
Elissa Redmiles, Lisa Maszkiewicz, Emily Hwang, Dhruv Kuchhal, Everest Liu, Miraida Morales, Denis Peskov, Sudha Rao, Rock Stevens, Kristina Gligorić, Sean Kross, Michelle Mazurek, Hal Daumé III
| Challenge: | Despite this, we lack a thorough understanding of how to validly measure readability at scale, especially for domain-specific texts. |
| Approach: | They present a comparison of the validity of well-known readability measures and introduce a novel approach to measure readability at scale. |
| Outcome: | The proposed approach addresses shortcomings of existing measures. |
Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features (2021.emnlp-main)
Copied to clipboard
| Challenge: | ML models with handcrafted features are linguistically explainable, expandable, and competent against the modern neural models. |
| Approach: | They propose to combine traditional ML models with ML transformers to improve readability assessment by 99% accuracy. |
| Outcome: | The proposed model achieves state-of-the-art (SOTA) accuracy on popular datasets. |
DOPA METER – A Tool Suite for Metrical Document Profiling and Aggregation (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Xiao et al., 2022) examines the behavior of written language in a metrical way. |
| Approach: | They propose a tool suite for the metrical investigation of written language that provides diagnostic means for its division into discourse categories, such as registers, genres, and style. |
| Outcome: | The proposed system provides means for scoring linguistic behavior at the lexical, syntactic and semantic dimension. |
CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation protocols for text generation suffer from rating inconsistencies . lexical overlap-based metrics align poorly with human judgments . |
| Approach: | They propose a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. |
| Outcome: | The proposed framework improves rating reliability by decomposing binary questions . it improves agreement across evaluator models by 0.45 and reduces score variance . human evaluation remains the gold standard, but it #, Equal contribution. |
Koala: An Index for Quantifying Overlaps with Pre-training Corpora (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models can be influenced by the frequency of overlap between pre-training corpora. |
| Approach: | They propose to search over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. |
| Outcome: | Koala is a searchable index over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. |
GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may encourage cherry-picking favored settings and for better results. |
| Approach: | They propose an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals that systematically evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
| Outcome: | The evaluation suite is built on top of OpenAI Evals and evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
CorefInst: Leveraging LLMs for Multilingual Coreference Resolution (2026.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for CR are encoder-only, decoder-based and asynchronous models. |
| Approach: | They propose a multilingual CR methodology which leverages decoder-only LLMs to handle overt and zero mentions. |
| Outcome: | The proposed model outperforms the leading multilingual CR model by 2 percentage points across all languages in the CorefUD v1.2 dataset. |
The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models (2020.emnlp-demos)
Copied to clipboard
Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, Ann Yuan
| Challenge: | Existing tools for modeling and understanding models are limited . existing tools can assist practitioners in understanding and evaluating models . |
| Approach: | They present an open-source platform for visualization and understanding of NLP models. |
| Outcome: | The language interpretability tool (lit) is an open-source platform for visualization and understanding of NLP models. |
Measuring What Matters: Evaluating Ensemble LLMs with Label Refinement in Inductive Coding (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are prone to inconsistencies and individual biases, limiting their reliability. |
| Approach: | They propose a framework that combines ensemble methods with code refinement methodology to address these challenges. |
| Outcome: | The proposed framework outperforms large language models and LLMs with a low-rank averaging and a moderator-based mechanism to simulate human consensus. |
Learning Syntactic Dense Embedding with Correlation Graph for Automatic Readability Assessment (2021.acl-long)
Copied to clipboard
| Challenge: | Existing deep learning models for automatic readability assessment discard linguistic features traditionally used for the task. |
| Approach: | They propose to incorporate linguistic features into machine learning models by learning syntactic dense embeddings based on linguistic feature extraction. |
| Outcome: | Experiments with six data sets of two proficiency levels show that the proposed model can perform better than existing models. |