Papers by Brian Roark
Spelling convention sensitivity in neural language models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Various long-distance dependencies have been investigated using neural language models. |
| Approach: | They examine whether large neural language models learn the long-distance dependency of British versus American spelling conventions . a large T5 language model does internalize consistency, but only with respect to observed lexical items . |
| Outcome: | The proposed model internalizes consistency with the training corpora, but only with respect to observed lexical items. |
Improving Informally Romanized Language Identification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Latin script is often used to informally write languages with non-Latin native scripts, resulting in high spelling variability. |
| Approach: | They propose to improve methods used to synthesize training sets to incorporate natural spelling variations into training sets. |
| Outcome: | The proposed method improves test F1 from the reported 74.7% (using a pretrained neural model) to 85.4% (using the linear classifier trained solely on synthetic data). |
What Kind of Language Is Hard to Language-Model? (P19-1)
Copied to clipboard
| Challenge: | a recent study suggests that language models perform poorly across languages. |
| Approach: | They propose a model that fits a paired-sample multiplicative mixed-effects model to obtain language difficulty coefficients from at least-pairwise parallel corpora. |
| Outcome: | The proposed model is able to handle missing data and is aware of inter-sentence variation. |
Finding Concept-specific Biases in Form–Meaning Associations (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to detect cross-linguistic associations are not effective, but their effects are minor. |
| Approach: | They propose a method to measure cross-linguistic associations by controlling for the influence of language family and geographic proximity within a large concept-aligned, cross-lingual lexicon. |
| Outcome: | The proposed method shows that it is small, but it is unsurprisingly small (less than 0.5% on average). |
Phonotactic Complexity and Its Trade-offs (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing measures of linguistic complexity are relatively coarse-see, for example, Moran and Blasi (2014) and 2 below for reviews. |
| Approach: | They propose to measure bits per phoneme using the negative log-probability of a word in a language model and a collection of 1016 basic concept words across 106 languages. |
| Outcome: | The proposed measure allows a cross-linguistic comparison of phonotactic complexity across languages. |
Structured abbreviation expansion in context (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Ad hoc abbreviations are commonly found in informal communication channels that favor shorter messages. |
| Approach: | They propose to reverse ad hoc abbreviations in context to recover normalized, expanded versions of abbrevated messages. |
| Outcome: | The proposed method can recover normalized, expanded abbreviations from text . it is similar to spelling correction, but requires more extensive work . |
Meaning to Form: Measuring Systematicity as Information (P19-1)
Copied to clipboard
| Challenge: | A longstanding debate in semiotics centers on the relationship between linguistic signs and their corresponding semantics: is there an arbitrary relationship between word forms and their meaning, or does some systematic phenomenon pervade? |
| Approach: | They propose to quantify the systematicity of the sign using mutual information and recurrent neural networks to examine 106 languages. |
| Outcome: | The proposed model reduces entropy in a word form conditioned on its semantic representation and recovers English examples of systematic affixes. |
Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset (2020.lrec-1)
Copied to clipboard
Brian Roark, Lawrence Wolf-Sonkin, Christo Kirov, Sabrina J. Mielke, Cibu Johny, Isin Demirsahin, Keith Hall
| Challenge: | a new resource is available for 12 South Asian languages that use the Latin script for text entry . the Latin-script system is not widely used in South Asian language writing, despite the Latin alphabet . |
| Approach: | They describe the Dakshina dataset, a new resource consisting of text in both the Latin and native scripts for 12 South Asian languages. |
| Outcome: | The Dakshina dataset includes text in both the Latin and native scripts for 12 languages . the authors provide baseline results on several tasks made possible by the dataset . |
Disambiguatory Signals are Stronger in Word-initial Positions (2021.eacl-main)
Copied to clipboard
| Challenge: | lexical studies of word processing and lexicon access provide evidence of preferred nature of word-initial versus word-final segments . conjecture that languages have evolved to provide more information earlier in words is based on existing methods . |
| Approach: | They propose to use a new method to assess the informativeness of word-initial versus word-final segments. |
| Outcome: | The proposed measures avoid the confounds found in existing methods. |
Are All Languages Equally Hard to Language-Model? (N18-2)
Copied to clipboard
| Challenge: | a fair comparison of language models is tricky because of the size of the corpora and the variability of orthographic systems. |
| Approach: | They propose a framework for fair cross-linguistic comparison of language models . they show that in some languages, textual expression is harder to predict with n-gram models compared to LSTM models based on translated text . |
| Outcome: | The proposed framework is based on translated text and language models on 21 languages. |
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages (2023.findings-emnlp)
Copied to clipboard
Sebastian Ruder, Jonathan Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David Adelani, Vera Axelrod, Isaac Caswell, Colin Cherry, Dan Garrette, Reeve Ingle, Melvin Johnson, Dmitry Panteleev, Partha Talukdar
| Challenge: | Existing datasets are often informed by established research directions in the NLP community. |
| Approach: | They propose a benchmark to evaluate the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks. |
| Outcome: | The proposed benchmark evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks. |
Extensions to Brahmic script processing within the Nisaba library: new scripts, languages and utilities (2022.lrec-1)
Copied to clipboard
| Challenge: | a brahmic script is used to record endangered languages such as Dogri and Bengali for low-resource languages such that do not require visual normalization. |
| Approach: | They propose to extend Brahmic script functionality within the Nisaba library of finite-state script normalization and processing utilities. |
| Outcome: | The proposed extensions extend coverage from the original ten scripts to an additional ten of South Asia and beyond, including some used to record endangered languages such as Dogri. |
Finite-state script normalization and processing utilities: The Nisaba Brahmic library (2021.eacl-demos)
Copied to clipboard
| Challenge: | a library for low-level processing of brahmic scripts is available for free. |
| Approach: | They propose an open-source library for efficient low-level processing of ten major South Asian Brahmic scripts. |
| Outcome: | The proposed library supports low-level processing of ten major south Asian Brahmic scripts. |
Criteria for Useful Automatic Romanization in South Asian Languages (2022.lrec-1)
Copied to clipboard
| Challenge: | a number of possible criteria for systems that transliterate South Asian languages are considered . romanization is the special case where the target script is the Latin script. |
| Approach: | They propose a set of criteria for systems that transliterate South Asian languages . criteria include fidelity to human linguistic behavior, processing utility for people, invertibility . they then propose several algorithms that address different criteria . |
| Outcome: | The proposed algorithms address linguistic considerations in the context of Brahmic scripts and languages that use them, such as Hindi and Malayalam. |