Papers by Marco Cognetta
Online Infix Probability Computation for Probabilistic Finite Automata (P19-1)
Copied to clipboard
| Challenge: | Probabilistic finite automata (PFAs) are statistical language models used in natural language processing. |
| Approach: | They develop an asymptotic algorithm to compute the infix probabilities of each prefix of a string from streaming data. |
| Outcome: | The proposed algorithm improves the infix probabilities of a weighted automata from streaming data. |
Parameter-Efficient Korean Character-Level Language Modeling (2023.eacl-main)
Copied to clipboard
| Challenge: | Character-level language modeling has been shown to perform well on highly agglutinative or morphologically rich languages while using only a small fraction of the parameters required by (sub)word models. |
| Approach: | They propose a “three-hot” embedding and decoding scheme that exploits the decomposability of Korean characters to model at the syllable level but using only jamo-level representations. |
| Outcome: | The proposed model reduces the embedding parameters by 99.6% and does not lose translation quality compared to the baseline model. |
SoftRegex: Generating Regex from Natural Language Descriptions using Softened Regex Equivalence (D19-1)
Copied to clipboard
| Challenge: | Existing models that generate semantically correct regular expressions from NLs are not yet fully understood. |
| Approach: | They propose a model that rewards reinforcement learning based on the semantic equivalence between two regular expressions. |
| Outcome: | The proposed model reduces training time and produces state-of-the-art results on three benchmark datasets. |
Two Counterexamples to Tokenization and the Noiseless Channel (2024.lrec-main)
Copied to clipboard
| Challenge: | Nevertheless, Rényi efficiency is not perfect and the metric is difficult to evaluate because training multiple tokenizers can be prohibitively expensive and takes days or weeks. |
| Approach: | They propose to use Rényi efficiency as an intrinsic mechanism to evaluate a tokenizer for NLP tasks without the expensive step of training multiple models with different tokenizers. |
| Outcome: | The proposed metric is better correlated to downstream model performance than a percentile frequency metric. |
Incremental Computation of Infix Probabilities for Probabilistic Finite Automata (D18-1)
Copied to clipboard
| Challenge: | Probabilistic finite automata are used to model distributions in natural language processing . a method that computes infix probabilities incrementally is proposed . |
| Approach: | They propose a method that computes infix probabilities incrementally for probabilistic finite automata . they propose to represent all the probabilities of matching strings as a series of transition matrix calculations . |
| Outcome: | The proposed method is theoretically faster than the previous method and better in practice. |
Distributional Properties of Subword Regularization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Subword regularization reduces the dependency on exact tokenizations, augments training corpus, and exposes model to unique contexts during training. |
| Approach: | They propose an algorithm to uniformly sample subword tokenizations to replace stochastic variants that are biased towards a small set of tokenization per word. |
| Outcome: | The proposed algorithm reduces the dependency on exact tokenizations and augments the training corpus. |