| Challenge: | Existing deep learning tools offer building blocks but training and building models takes time and knowledge. |
| Approach: | They propose to make available LSTM language models trained on Dutch texts and English benchmarks. |
| Outcome: | The proposed model can be used to test the perplexity, predict the next word(s), re-score hypotheses or generate debugging files for interpolation with n-gram models. |
Similar Papers
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use (2025.findings-emnlp)
Copied to clipboard
Junjie Ye, Yilong Wu, Sixian Li, Yuming Yang, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan, Zhengyin Du
| Challenge: | a new approach to training large language models (LLMs) overlooks task-specific characteristics in tool use, leading to performance bottlenecks. |
| Approach: | They propose a task-feature-based framework that mitigates the effects of suboptimal training data . they use a dataset to train large-scale LLMs and a reward mechanism tailored to error categories . |
| Outcome: | The proposed framework matches or surpasses open- and closed-source LLMs in tool-use performance using only 1,217 training data points. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
A Deep Dive into Word Sense Disambiguation with LSTM (C18-1)
Copied to clipboard
| Challenge: | LSTM-based language models have been shown effective in Word Sense Disambiguation (WSD) but neither the training data nor the source code was released. |
| Approach: | They propose to use LSTM-based language models to perform Word Sense Disambiguation (WSD) using openly available datasets and software. |
| Outcome: | The proposed method returned state-of-the-art performance in several benchmarks, but neither the training data nor the source code were released. |
On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing decoder-based pre-trained language models demonstrate excellent multilingual capabilities, but it is unclear how they handle multilingualism. |
| Approach: | They propose to examine the neuron-level internal behavior of decoder-based PLMs by finding neurons that fire “uniquely for each language” within decoded PLM models. |
| Outcome: | The proposed models fire “uniquely for each language” and show that language-specific neurons are unique, with a slight overlap (5%) between languages. |
xLM: A Python Package for Non-Autoregressive Language Models (2026.eacl-demo)
Copied to clipboard
| Challenge: | Autoregressive language models generate text sequentially from left to right by adding one token at a time. |
| Approach: | They propose a python package that provides a suite of small non-autoregressive language models that can be used by researchers. |
| Outcome: | The proposed package makes implementing small non-autoregressive language models faster and provides a suite of pre-trained models that can be used by the research community. |
LMentry: A Language Model Benchmark of Elementary Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models are evaluated via perplexity or performance on downstream tasks, but these benchmarks are too complex and difficult to inspect. |
| Approach: | They propose a benchmark that focuses on 25 tasks that humans are expected to perform perfectly, such as writing a sentence containing a specific word or identifying which words in a list belong to a certain category. |
| Outcome: | The proposed benchmarks show that large language models are performing better than previous benchmarks. |
Evaluating Language Models as Synthetic Data Generators (2025.acl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, Graham Neubig
| Challenge: | Prior studies have focused on developing effective data generation methods, but lack systematic comparison of different LMs as data generators in a unified setting. |
| Approach: | They propose to use a benchmark to compare language models' data generation abilities against a set of standardized settings and metrics. |
| Outcome: | The proposed benchmark provides standardized settings and metrics to evaluate LMs’ data generation abilities. |
T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual text classification leverage text classifiers trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning. |
| Approach: | They propose to combine a neural machine translator and a text classifier trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning. |
| Outcome: | The proposed approach significantly improves over a baseline approach. |
Controlled Language Generation for Language Learning Items (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in pre-trained language models have resulted in success in generating fluent English text. |
| Approach: | They propose to employ natural language generation to rapidly generate English language items . they experiment with deep pretrained models and develop methods for controlling items for factors relevant in language learning . |
| Outcome: | The proposed framework shows high grammatically scores for all models and higher complexity over baseline models. |