Papers by Sami Virpioja
Morfessor EM+Prune: Improved Subword Segmentation with Expectation Maximization and Pruning (2020.lrec-1)
Copied to clipboard
| Challenge: | Subword segmentation is a standard preprocessing step in many neural approaches to natural language processing. |
| Approach: | They propose to train a unigram subword model using a recursive algorithm and lexicon pruning algorithm. |
| Outcome: | The proposed method improves on the original training algorithm and improves morphological segmentation accuracy. |
OpusFilter: A Configurable Parallel Corpus Filtering Toolbox (2020.acl-demos)
Copied to clipboard
| Challenge: | OpusFilter is a toolbox for filtering parallel corpora using noisy training data. |
| Approach: | They propose a toolbox for filtering parallel corpora with heuristic filters, language identification libraries, character-based language models and word alignment tools. |
| Outcome: | The proposed tool outperforms a similar tool on a Finnish-English news translation task using noisy web crawls. |
OpusTools and Parallel Corpus Diagnostics (2020.lrec-1)
Copied to clipboard
| Challenge: | Currently OPUS contains 57 released corpora covering over 700 languages and language variants creating more than 70,000 bitexts in the sense of aligned language pairs across all corporata. |
| Approach: | They introduce OpusTools, a package for downloading and processing parallel corpora in OPUS . the package implements tools for accessing compressed data in their archived release format . they show how they can be used in parallel corpus creation and data diagnostics . |
| Outcome: | The proposed tools can be used in parallel corpus creation and data diagnostics. |