Papers with fastText
A Study on Using Semantic Word Associations to Predict the Success of a Novel (2021.starsem-1)
Copied to clipboard
Syeda Jannatus Saba, Biddut Sarker Bijoy, Henry Gorelick, Sabir Ismail, Md Saiful Islam, Mohammad Ruhul Amin
| Challenge: | Existing methods for book success prediction are not effective. |
| Approach: | They propose to represent a book as a spectrum of concepts based on the association score between its content embedding and a global embeddment for a set of semantically linked word clusters. |
| Outcome: | The proposed method outperforms the previous methods for book success prediction. |
Intrinsic Probing through Dimension Selection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on probing for linguistic structure in word embeddings has focused on intrinsic probing, but what these representations encode about linguistic structures remains unclear. |
| Approach: | They propose a framework that allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
| Outcome: | The proposed framework allows us to determine whether linguistic information in word embeddings is dispersed or focal. |
Exploring Semantic Spaces for Detecting Clustering and Switching in Verbal Fluency (2022.coling-1)
Copied to clipboard
| Challenge: | Existing evaluations of word/concept representations on verbal fluency tasks rely on human annotations of clusters and switches between sub-categories. |
| Approach: | They analyze word/concept representations in an experimental verbal fluency dataset . they find that ConceptNet embeddings outperforms other semantic representations . |
| Outcome: | The proposed method outperforms other semantic representations by a large margin. |
Magnitude: A Fast, Efficient Universal Vector Embedding Utility Package (D18-2)
Copied to clipboard
| Challenge: | Magnitude is an open source Python package that performs common operations up to 6,000 times faster than Gensim. |
| Approach: | They present a Python tool for utilizing vector embeddings that performs common operations up to 6,000 times faster than Gensim. |
| Outcome: | The Magnitude package performs common operations up to 6,000 times faster than Gensim and introduces several novel features for improved robustness like out-of-vocabulary lookups. |
Finnish Hate-Speech Detection on Social Media Using CNN and FinBERT (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing tools to identify hate posts from social media are limited in the field of online hate speech detection. |
| Approach: | They propose to use finBERT to generate a Finnish hate speech dataset . finBERt has a 91.7% accuracy and 90.8% F1 score value, they say . |
| Outcome: | The proposed model outperforms state-of-the-art models in Finnish and other languages. |
Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion (P19-1)
Copied to clipboard
| Challenge: | Existing speech recognition systems are built at individual, isolated utterance level to make building systems computationally feasible. |
| Approach: | They propose to use text-based external word and/or sentence embeddings to integrate conversational context information into a single neural network model. |
| Outcome: | The proposed model outperforms standard end-to-end speech recognition models on the Switchboard conversational speech corpus and improves word error rate with better conversational-context representation. |
Probing as Quantifying Inductive Bias (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained contextual representations have led to performance improvements on downstream tasks. |
| Approach: | They propose a Bayesian framework that quantifies the amount of inductive bias that the representations encode on a specific task. |
| Outcome: | The proposed framework alleviates many problems found in probing and can offer better inductive bias than BERT. |
Evaluation of Croatian Word Embeddings (L18-1)
Copied to clipboard
| Challenge: | Currently, research is focusing mostly on English. |
| Approach: | They propose to use word analogy datasets to evaluate word similarities in Croatian . they use Word2Vec and FastText to create word analogies from highdimensional space . |
| Outcome: | The proposed datasets show that word embeddings are able to capture the syntactic and semantic relationship between words. |
Evaluating the Impact of Sub-word Information and Cross-lingual Word Embeddings on Mi’kmaq Language Modelling (2020.lrec-1)
Copied to clipboard
| Challenge: | Mi'kmaq is an Indigenous language spoken primarily in Eastern Canada. |
| Approach: | They consider n-gram and RNN language models for Mi'kmaq and use them to investigate their performance. |
| Outcome: | The proposed model performs better than word-level models, but does not improve over word-based models. |
Multilingual Constituency Parsing with Self-Attention and Pre-Training (P19-1)
Copied to clipboard
| Challenge: | a range of pre-training conditions can be used for constituency parsing, but large model sizes make it expensive to train separate models for each language. |
| Approach: | They compare the benefits of no pre-training, fastText, ELMo, and BERT for English . they also find that pre- training is beneficial across all 11 languages tested . |
| Outcome: | The proposed model outperforms fastText, ELMo, and BERT for English . but large model sizes make it expensive to train separate models for each language . |
Task-adaptive Pre-training of Language Models with Word Embedding Regularization (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models acquire domain-independent knowledge through pre-training with massive textual resources. |
| Approach: | They propose a task-adaptive pre-training process that makes static embeddings close to the word embedds obtained in the target domain. |
| Outcome: | The proposed process improves on BioASQ and SQuAD when the pre-training corpora were not dominated by indomain data. |
LexFit: Lexical Fine-Tuning of Pretrained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Transformer-based language models implicitly store a wealth of lexical semantic knowledge, but it is non-trivial to extract that knowledge effectively from their parameters. |
| Approach: | They propose to expose and enrich lexical knowledge from transformer-based language models to serve as effective decontextualized word encoders even when fed input words "in isolation" |
| Outcome: | The proposed model outperforms standard static WEs and vanilla LMs in lexical tasks over four established tasks in 8 languages. |
HeLI-OTS, Off-the-shelf Language Identifier for Text (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing off-the-shelf language identification tools favor widely used languages, but Heli-OTS can be used to identify a large group of languages. |
| Approach: | They introduce an off-the-shelf text language identification tool using the HeLI method . they compare the He LI-OTS language identifier with fastText on two different data sets . |
| Outcome: | The proposed language identification tool is compared with fastText on two different data sets. |
Facilitating Corpus Usage: Making Icelandic Corpora More Accessible for Researchers and Language Users (2020.lrec-1)
Copied to clipboard
| Challenge: | Gigaword corpus is a large text corpus used in natural language processing . large corpora are needed to achieve better performance in the field of NLP . |
| Approach: | They propose a set of tools to facilitate the use of the Icelandic Gigaword Corpus . they provide n-grams based on the corpus, and a variety of pre-trained word embeddings models . |
| Outcome: | The proposed tools facilitate the use of the Icelandic Gigaword corpus in the field of Natural Language Processing and other fields. |
Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Pre-trained language models provide the foundations for state-of-the-art performance across a wide range of natural language processing tasks, including text classification. |
| Approach: | They compare the performance of a linear classifier based on word embeddings with a pre-trained language model, i.e., BERT, across a wide range of datasets and classification tasks. |
| Outcome: | The proposed method outperforms baselines in standard datasets with large training sets, but in settings with small training datasets it performs better. |
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data (2020.lrec-1)
Copied to clipboard
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, Edouard Grave
| Challenge: | Pre-training text representations have led to significant improvements in many areas of natural language processing. |
| Approach: | They propose a pipeline to extract monolingual datasets from Common Crawl . pipeline follows data processing introduced in fastText that deduplicates documents . |
| Outcome: | The proposed pipeline performs standard document deduplication and language identification similar to the pipeline introduced in fastText and a filtering step to select documents close to high quality corpora like Wikipedia. |
Czech Historical Named Entity Corpus v 1.0 (2020.lrec-1)
Copied to clipboard
| Challenge: | a lack of annotated historical data for named entity recognition is an obstacle to research in this area. |
| Approach: | They propose to create an annotated corpus for named entity recognition in historical documents . they define domain-specific named entity types and create an annotation manual . |
| Outcome: | The proposed corpus is available for research and is available to download . it is the first annotated historical corpus for named entity recognition (NER) |
Beyond Distribution: Investigating Language Models’ Understanding of Sino-Korean Morphemes (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Transformer-based language models can learn compositional morphology of SK morphemes . morphological models trained on Hangul text can learn SK, but performance is based on frequency of words . |
| Approach: | They investigate whether Transformer-based language models can learn compositional morphology of Sino-Korean morphemes. |
| Outcome: | The proposed models learn the compositional morphology of SK morphemes from real and fake pairs. |
CBOW-tag: a Modified CBOW Algorithm for Generating Embedding Models from Annotated Corpora (2020.lrec-1)
Copied to clipboard
| Challenge: | Using word2vec, we train distributional semantic models that predict a word from the context or vice versa. |
| Approach: | They propose a modified version of the CBOW algorithm implemented in the fastText framework that includes the representation of original word forms and their annotation at the same time. |
| Outcome: | The proposed model can answer questions such as What do we eat?, What can we do with a skeleton?, etc. |
FastSpell: The LangId Magic Spell (2024.lrec-main)
Copied to clipboard
| Challenge: | Language identification is a crucial component in the automated production of language resources. |
| Approach: | They propose a language identifier that combines fastText and Hunspell to give a second opinion before deciding which language to assign to a text. |
| Outcome: | The proposed language identifier is based on a pre-trained language identifier and a spell checker. |
Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary? (2025.coling-main)
Copied to clipboard
Adam Nohejl, Frederikus Hudi, Eunike Andriani Kardinata, Shintaro Ozaki, Maria Angelica Riera Machin, Hongyu Sun, Justin Vasselli, Taro Watanabe
| Challenge: | Word frequency is a key variable in psycholinguistics, useful for modeling human familiarity with words . a recent study shows that frequency from YouTube subtitles is comparable to and often better than the best available resources. |
| Approach: | They use YouTube subtitles to construct frequency norms for five languages . they find they are comparable to and often better than the best currently available resources . |
| Outcome: | The proposed method improves on the best currently available resources for Chinese, English, Indonesian, Japanese, and Spanish. |
Word Sense Disambiguation for 158 Languages using Word Embeddings Only (2020.lrec-1)
Copied to clipboard
Varvara Logacheva, Denis Teslenko, Artem Shelmanov, Steffen Remus, Dmitry Ustalov, Andrey Kutuzov, Ekaterina Artemova, Chris Biemann, Simone Paolo Ponzetto, Alexander Panchenko
| Challenge: | Existing methods of disambiguation of word senses are based on knowledge bases, taxonomies, and other externally built resources. |
| Approach: | They propose a method that takes a pre-trained word embedding model and induces a fully-fledged word sense inventory for 158 languages. |
| Outcome: | The proposed model is based on a pre-trained word embedding model and induces a fully-fledged word sense inventory in 158 languages. |