Papers by Sjoerd Steenkiste
A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Psychologists have documented several ways in which humans’ inferences deviate from the rules of logic. |
| Approach: | They focus on syllogisms, which are inferences from two simple premises, and show that larger models are more logical than smaller ones. |
| Outcome: | The results show that language models often mimic human biases, but overcome them in some cases. |
Benchmarking Vision Language Models for Cultural Understanding (2024.emnlp-main)
Copied to clipboard
Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Steenkiste, Lisa Hendricks, Karolina Stanczak, Aishwarya Agrawal
| Challenge: | Recent multimodal vision-language models have shown impressive performance in tasks such as image-to-text generation, visual question answering, and image captioning. |
| Approach: | They propose a visual question-answering benchmark to assess VLMs' cultural understanding of various facets of culture from 11 countries across 5 continents. |
| Outcome: | The visual question-answering benchmark aims to assess VLMs' cultural understanding across regions. |
The Impact of Depth on Compositional Generalization in Transformer Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we test the hypothesis that deeper transformers generalize more compositionally. |
| Approach: | They propose to add layers to transformers to generalize more compositionally . they propose to fine-tune the models so that the total number of parameters is constant . |
| Outcome: | The proposed model generalizes more compositionally than shallower models, but returns diminish . the proposed model can be made shallower without sacrificing performance . |