Papers by Richard Johansson
Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Augmenting language models with a retrieval mechanism has been shown to improve performance while keeping the number of parameters low. |
| Approach: | They propose to augment language models with a retrieval mechanism by replacing semantic retrieval with BM25 . they find that the model's performance is better explained by surface-level similarities, they say . |
| Outcome: | The proposed method reduces perplexity and lowers the number of parameters while keeping the number low. |
Transformer-based Swedish Semantic Role Labeling through Transfer Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Semantic Role Labeling (SRL) is a task in natural language understanding where the goal is to extract semantic roles for a given sentence. |
| Approach: | They propose to build a Transformer-based SRL system for Swedish by exploring multilingual and cross-lingual transfer learning methods and leveraging the Swedish FrameNet resource. |
| Outcome: | The proposed model outperforms two different cross-lingual transfer models and shows that the multilingual learning outperformed the other models. |
Training a Swedish Constituency Parser on Six Incompatible Treebanks (2020.lrec-1)
Copied to clipboard
| Challenge: | Syntactic parsing is a widely used intermediate step in several natural language processing tasks. |
| Approach: | They propose to use a function-tagged constituent treebank for Swedish which includes discontinuous constituents to improve the accuracy. |
| Outcome: | The proposed parser can be trained on additional treebanks that use other annotation models. |
Benchmarking Debiasing Methods for LLM-based Parameter Estimates (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are expensive yet powerful ways to annotate text, and can be inconsistent when compared with experts. |
| Approach: | They propose to combine LLM annotations with a limited number of expensive expert annotations to produce valid estimates. |
| Outcome: | The proposed methods produce consistent estimates under theoretical assumptions, but they are not comparable across finite datasets. |
What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge (2022.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation methods to measure what language models learn from multimodal training are lacking. |
| Approach: | They propose two evaluation tasks to measure commonsense knowledge in language models by using visual data to evaluate multimodal models and unimodal baselines. |
| Outcome: | The proposed evaluation tasks show that training on a visual modality improves on the visual commonsense knowledge in language models. |
What Happens to a Dataset Transformed by a Projection-based Concept Removal Method? (2024.lrec-main)
Copied to clipboard
| Challenge: | A number of recent methods have used linear projections to remove information about a concept from a language representation. |
| Approach: | They propose to use linear projections to remove a concept from a language representation to create a transformed set of word embeddings. |
| Outcome: | The proposed methods inject strong statistical dependencies into the transformed datasets. |
The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are useful interfaces to factual knowledge, but their usefulness is limited by their tendency to deliver inconsistent answers to semantically equivalent questions. |
| Approach: | They evaluate the effectiveness of up-scaling and augmenting the LM with a passage retrieval database to reduce inconsistency. |
| Outcome: | The proposed models reduce inconsistency but retrieval augmentation is more efficient. |
On the Generalization Ability of Retrieval-Enhanced Transformers (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent work on retrieval-augmented language models has shown impressive results . performance gains from retrieval to a large extent originate from overlapping tokens between the database and test data, suggesting less of non-trivial generalization than previously assumed. |
| Approach: | They propose to off-load memory from trainable weights to a retrieval database and compare it to larger models with a larger model. |
| Outcome: | The proposed model outperforms GPT-3 and Jurassic-1 on the Pile at 4% of the model parameters. |
Conceptualizing Treatment Leakage in Text-based Causal Inference (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to control for text-based confounders rely on assumption that there is no treatment leakage . prior literature has assumed that documents only contain information about confounder, but not about treatment assignment. |
| Approach: | They define the treatment leakage problem and propose methods to mitigate it . they remove treatment-related signal from text in a pre-processing step . |
| Outcome: | The proposed method can mitigate the problem of treatment leakage by removing the treatment-related signal from the text. |
Deciphering the Interplay of Parametric and Non-parametric Memory in Retrieval-augmented Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Generative language models struggle with specialized knowledge that is discussed less frequently on the web. |
| Approach: | They propose to use a model which decides between parametric and non-parametric knowledge to investigate how it uses the information from the context. |
| Outcome: | The proposed model can choose between parametric and non-parametric information, but relies more on context than parametric knowledge. |
CUB: Benchmarking Context Utilisation Techniques for Language Models (2026.acl-long)
Copied to clipboard
Lovisa Hagström, Youna Kim, Haeun Yu, Sang-goo Lee, Richard Johansson, Hyunsoo Cho, Isabelle Augenstein
| Challenge: | Existing language models (LMs) can be distracted by irrelevant contexts or ignore relevant information that contradicts outdated parametric memory. |
| Approach: | They develop a benchmark to help diagnose CMTs under diverse noisy context conditions within retrieval-augmented generation (RAG) they find that most existing CMT struggle to handle the full spectrum of context types encountered in real-world RAG scenarios. |
| Outcome: | The proposed benchmark compares seven state-of-the-art methods across three datasets and tasks, and shows that many lack the robustness needed to handle the full spectrum of context types encountered in real-world RAG scenarios. |
Fact Recall, Heuristics or Pure Guesswork? Precise Interpretations of Language Models for Fact Completion (2025.findings-acl)
Copied to clipboard
| Challenge: | Language models (LMs) can make a correct prediction based on many possible signals in a prompt, but not all corresponding to recall of factual associations. |
| Approach: | They propose a model-specific recipe for constructing datasets with examples of four different prediction scenarios: generic language modeling, guesswork, heuristics recall and exact fact recall. |
| Outcome: | The proposed model-specific recipe yields distinct results for each scenario. |
How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input? (2022.coling-1)
Copied to clipboard
| Challenge: | Current language models have been criticised for learning language from text alone without connection between words and their meaning. |
| Approach: | They propose to train models on more sources than text to provide the lacking connection between words and their meanings. |
| Outcome: | The proposed model adaptation methods perform differently for different models and unimodal model counterparts perform on par with the VL models regardless of adaptation. |