Evaluating Historical Text Normalization Systems: How Well Do They Generalize? (N18-2)
Copied to clipboard
| Challenge: | Historical text normalization systems aim to convert historical wordforms to their modern equivalents . many of these systems have been developed and tested on a single language . |
| Approach: | They propose to use a nave baseline system to evaluate historical text normalization systems . they show that the models generalize well to unseen words in tests on five languages . |
| Outcome: | The proposed models generalize well to unseen words on five languages, but provide no clear benefit over the nave baseline. |
Similar Papers
A Large-Scale Comparison of Historical Text Normalization Systems (N19-1)
Copied to clipboard
| Challenge: | a large study of historical text normalization is done on eight languages . there is no consensus on the state-of-the-art approach to normalization . |
| Approach: | They present a large study of historical text normalization done on eight languages . they evaluate four different systems based on supervised learning on datasets from eight different languages based in the literature . |
| Outcome: | The proposed methods are based on supervised learning and are available online. |
Semi-supervised Contextual Historical Text Normalization (2020.acl-main)
Copied to clipboard
| Challenge: | Historical text normalization is the task of mapping historical word forms to their modern counterparts. |
| Approach: | They propose to use a generative normalization model to obtain contextualization from the target-side language model. |
| Outcome: | et al., 2018) show that the most effective approach reduces manual normalization time and manual training costs. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
Out-of-Distribution Generalization in Natural Language Processing: Past, Present, and Future (2023.emnlp-main)
Copied to clipboard
Linyi Yang, Yaoxian Song, Xuan Ren, Chenyang Lyu, Yidong Wang, Jingming Zhuo, Lingqiao Liu, Jindong Wang, Jennifer Foster, Yue Zhang
| Challenge: | Existing literature on the generalization of machine learning models to out-of-distribution data is lacking. |
| Approach: | They propose to present the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
| Outcome: | The proposed survey provides the first comprehensive review of recent progress, methods, and evaluations on the generalization challenge from an OOD perspective in natural language understanding. |
Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Lexical normalization research has sought to tackle the challenge of processing informal expressions in user-generated text. |
| Approach: | They focus on Japanese normalization and developing methods based on state-of-the-art pre-trained models . |
| Outcome: | The proposed methods achieve high accuracy and efficiency across multiple evaluation perspectives. |
First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | a new system trained on well over a trillion words smashes the state of the art by a margin previously thought impossible. |
| Approach: | They argue that disparities in scale are transient and researchers can work to reduce them . they argue that data, rather than hardware, is still a bottleneck for many applications . |
| Outcome: | a new system trained on well over a trillion words smashes the state of the art by a margin previously thought impossible. |
A Detailed Evaluation of Neural Sequence-to-Sequence Models for In-domain and Cross-domain Text Simplification (L18-1)
Copied to clipboard
| Challenge: | Xu et al., 2016) show that a simple neural architecture can be efficiently used for in-domain and cross-domain text simplification. |
| Approach: | They evaluate neural sequence-to-sequence models for text simplification on Wikipedia and Newsela datasets. |
| Outcome: | The proposed model can generalize across corpora and overcome challenges when tested on Wikipedia and Newsela datasets. |
Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges (2025.acl-long)
Copied to clipboard
Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, Barbara Plank
| Challenge: | linguistics studies how context influences meaning of language and how people use it to convey implied meanings, emotions, and intentions. |
| Approach: | They analyze task designs, data collection methods, evaluation approaches and their relevance to real-world applications. |
| Outcome: | The findings highlight emerging trends, challenges, and gaps in existing benchmarks . the findings will contribute to more nuanced and context-aware NLP models . |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Few-Shot and Zero-Shot Learning for Historical Text Normalization (D19-61)
Copied to clipboard
| Challenge: | Historical text normalization often relies on small training datasets. |
| Approach: | They evaluate 63 multi-task learning configurations for sequence-to-sequence-based historical text normalization across ten datasets from eight languages. |
| Outcome: | The proposed learning architecture outperforms the simple, but strong identity baseline. |