Papers by Ioannis Konstas
In Layman’s Terms: Semi-Open Relation Extraction from Scientific Texts (2020.acl-main)
Copied to clipboard
| Challenge: | Information Extraction (IE) systems extract only a fraction of the information captured, and Open IE systems do not perform well on the long and complex sentences encountered in scientific texts. |
| Approach: | They propose to use Focused Open Biological Information Extraction (FOBIE) to train a narrow scientific IE system to extract trade-off relations and arguments that are central to biology texts. |
| Outcome: | The proposed system extracts trade-off relations and arguments that are central to biology texts. |
Imagining Grounded Conceptual Representations from Perceptual Information in Situated Guessing Games (2020.coling-main)
Copied to clipboard
Alessandro Suglia, Antonio Vergari, Ioannis Konstas, Yonatan Bisk, Emanuele Bastianelli, Andrea Vanzo, Oliver Lemon
| Challenge: | Existing models fail to learn multi-modal representations, relying on category labels at inference time. |
| Approach: | They propose a "imagination" module that learns context-aware and category-awful latent embeddings without relying on category labels at inference time. |
| Outcome: | The imagination module outperforms state-of-the-art competitors by 8.26% gameplay accuracy in the CompGuessWhat?! benchmark. |
Better Conversations by Modeling, Filtering, and Optimizing for Coherence and Diversity (D18-1)
Copied to clipboard
| Challenge: | Existing encoder-decoder models for open domain dialogue generate generic, uninformative, and non-coherent responses. |
| Approach: | They propose to introduce a measure of coherence as the GloVe embedding similarity between dialogue context and generated response to improve output diversity. |
| Outcome: | The proposed model improves on the OpenSubtitles corpus in terms of BLEU score and diversity metrics. |
OTTers: One-turn Topic Transitions for Open-Domain Dialogue (2021.acl-long)
Copied to clipboard
| Challenge: | a mixed-initiative dialogue system is often purely responsive, make abrupt transitions, or fail to take initiative. |
| Approach: | They propose a task to generate a "bridging" utterance connecting a new topic to the previous conversation turn. |
| Outcome: | The proposed task generates a "bridging" utterance connecting a new topic to the previous topic. |
Voices in a Crowd: Searching for clusters of unique perspectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Proposed solutions aim to capture minority perspectives by either modelling annotator disagreements or grouping annotators based on shared metadata. |
| Approach: | They propose a framework that trains models without encoding annotator metadata and creates clusters of similar opinions, that are called voices. |
| Outcome: | The proposed framework captures minority perspectives based on demographic factors in two distinct datasets while also capturing majority perspectives. |
SEQˆ3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression (N19-1)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models are currently the dominant approach in natural language processing tasks, but require massive parallel corpora. |
| Approach: | They propose a sequence-to-sequence-tosequnce autoencoder with words as latent variables . they apply the model to unsupervised abstractive sentence compression . |
| Outcome: | The proposed model achieves promising results in unsupervised sentence compression on benchmark datasets. |
Re-examining Sexism and Misogyny Classification with Annotator Attitudes (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets for content moderation fail to capture plurality of possible annotator perspectives or ensure representation of affected groups. |
| Approach: | They examine the relationship between annotator identities and attitudes and the responses they give to two GBV labelling tasks. |
| Outcome: | The results show that higher Right Wing Authoritarianism scores are associated with a higher propensity to label text as sexist . higher scores are also associated with negative attitudes towards sexism and neosexist attitudes . |
MiRANews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Current news summarization systems often contain 'extrinsic hallucinations', i.e. facts that are not present in the source document, which are often derived via world knowledge. |
| Approach: | They propose to use multiple supplementary resource documents to assist the task by pairing a single document with a human authored summary as the summary. |
| Outcome: | The proposed model reduces 55% of hallucinations when compared to single-document summarization models trained on the main article only. |
The Larger they are, the Harder they Fail: Language Models do not Recognize Identifier Swaps in Python (2023.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used for programming tasks but lack a deep understanding of the content they manipulate. |
| Approach: | They show that LLMs fail to correctly generate correct Python code when default function names are swapped . they also show that they become more confident in their incorrect predictions as the model size increases . |
| Outcome: | The proposed models fail to generate correct Python code when default function names are swapped, and become more confident in their incorrect predictions as the model size increases. |
Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) for data-to-text generation produce inaccurate outputs if labels are ambiguous or incomplete, which is often the case in D2T datasets. |
| Approach: | They propose to use a dataset to descib a relation between two entities using relation labels to train pretrained language models. |
| Outcome: | The proposed models are robust to generalizing to out-of-domain domains on a dataset for descibing a relation between two entities. |
The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answering (2023.findings-acl)
Copied to clipboard
Sabrina Chiesurin, Dimitris Dimakopoulos, Marco Antonio Sobrevilla Cabezudo, Arash Eshghi, Ioannis Papaioannou, Verena Rieser, Ioannis Konstas
| Challenge: | Empirical studies of dialogue have shown that people use different kinds of context-dependent linguistic behavior to indicate grounding, including use of fragments, ellipsis and pronominal reference. |
| Approach: | They propose to use open-domain question answering systems as test-bed for task based dialog generation and compare open- and closed-book models to test their hypothesis. |
| Outcome: | The proposed model parrots user input while providing an unfaithful response. |
Multitask Multimodal Prompted Training for Interactive Embodied Task Completion (2023.emnlp-main)
Copied to clipboard
Georgios Pantazopoulos, Malvina Nikandrou, Amit Parekh, Bhathiya Hemanthage, Arash Eshghi, Ioannis Konstas, Verena Rieser, Oliver Lemon, Alessandro Suglia
| Challenge: | Embodied MultiModal Agent (EMMA) is a unified encoder-decoder model that reasons over images and trajectories and casts action prediction as multimodal text generation. |
| Approach: | They propose an Embodied MultiModal Agent (EMMA) that uses a unified encoder-decoder model that reasons over images and trajectories and casts action prediction as multimodal text. |
| Outcome: | The proposed model performs on par with similar models on several VL benchmarks and sets a new state-of-the-art success rate on the Dialog-guided Task Completion (DTC) benchmark. |
CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent Vision and Language models have shown impressive performance across benchmarks . however, frontier models lack cultural awareness and can affect global cultural diversity . |
| Approach: | They propose a visual question answering benchmark to probe the knowledge of culture-specific concepts and evaluate the capacity for cultural adaptation through contextual information. |
| Outcome: | The proposed model shows large performance disparities between culture-specific and common concepts in the parametric setting. |
Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating generalisation capabilities of multimodal models based solely on performance on out-of-distribution data fails to capture their true robustness . proposed framework examines the role of instructions and inputs in generalisation abilities of such models, considering architectural design, input perturbations across language and vision modalities, and increased task complexity. |
| Approach: | They propose a framework that examines the role of instructions and inputs in the generalisation abilities of multimodal models, considering architectural design, input perturbations across language and vision modalities, and increased task complexity. |
| Outcome: | The proposed framework examines the role of instructions and inputs in the generalisation abilities of multimodal models, considering architectural design, input perturbations across language and vision modalities, and increased task complexity. |
Fact-based Content Weighting for Evaluating Abstractive Summarisation (2020.acl-main)
Copied to clipboard
| Challenge: | Abstractive summarisation is notoriously hard to evaluate since word-overlap-based metrics are insufficient. |
| Approach: | They propose a new evaluation metric which is based on fact-level content weighting, relating the facts of the document to the facts in the summary. |
| Outcome: | The proposed evaluation metric is highly correlated to human perception and compares favourably to the recent manual highlight-based metric of Hardy et al. |
CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning (2020.acl-main)
Copied to clipboard
Alessandro Suglia, Ioannis Konstas, Andrea Vanzo, Emanuele Bastianelli, Desmond Elliott, Stella Frank, Oliver Lemon
| Challenge: | Approaches to Grounded Language Learning focus on a single task-based final performance measure which may not depend on desirable properties of the learned hidden representations. |
| Approach: | They propose an evaluation framework for Grounded Language Learning with Attributes based on three sub-tasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; and 3) Zero-shot evaluation. |
| Outcome: | The proposed framework evaluates the quality of learned representations with respect to attribute grounding. |
A Scientific Information Extraction Dataset for Nature Inspired Engineering (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing approaches to extract relevant biological information from scientific literature are difficult and require domain-specific knowledge. |
| Approach: | They describe a dataset of 1,500 manually-annotated sentences that express domain-independent relations between central concepts in a scientific biology text. |
| Outcome: | The proposed dataset allows for training and evaluation of Relation Extraction algorithms that aim for coarse-grained typing of scientific biological documents, enabling a high-level filter for engineers. |
History for Visual Dialog: Do we really need it? (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that dialog-based interaction grounded in visual information is not as effective as previous VQA tasks because of its dialog history. |
| Approach: | They propose a visual dialogue subset which explicitly encodes dialog history and a NDCG benchmark of 63%. |
| Outcome: | The proposed subset (VisdialConv) of the VisdialVal set achieves state-of-the-art performance on 72 % of the data. |
Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating Large Language Models (LLMs) on reasoning benchmarks demonstrates their ability to solve compositional questions. |
| Approach: | They investigate the transitive reasoning capabilities of two distinct LLM architectures, LLaMA 2 and Flan-T5, by manipulating facts within two compositional datasets: QASC and Bamboogle. |
| Outcome: | The proposed models leverage word/phrase overlaps across sections of test input, models’ inherent knowledge during pre-training or fine-tuning, and names of entities. |
AggGen: Ordering and Aggregating while Generating (2021.acl-long)
Copied to clipboard
| Challenge: | AggGen is a data-to-text model which re-introduces two explicit sentence planning stages into neural data- to-text systems: input ordering and input aggregation. |
| Approach: | AggGen re-introduces two explicit sentence planning stages into neural data-to-text systems: input ordering and input aggregation. |
| Outcome: | AggGen is a data-to-text model which re-introduces two explicit sentence planning stages into neural data- to-text systems: input ordering and input aggregation. |
AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding (2024.findings-emnlp)
Copied to clipboard
Alessandro Suglia, Claudio Greco, Katie Baker, Jose Part, Ioannis Papaioannou, Arash Eshghi, Ioannis Konstas, Oliver Lemon
| Challenge: | Current Vision-Language Models (VLMs) focus on third-person view videos, neglecting the richness of egocentric perceptual experience. |
| Approach: | They propose to use the Egocentric Video Understanding Dataset (EVUD) to train VLMs on video captioning and question answering tasks specific to egocentric videos. |
| Outcome: | The proposed model outperforms open-source models including strong Socratic models using GPT-4 as a planner by 3.6% and outperformed Claude 3 and Gemini Pro Vision 1.0. |
Findings of the Third Workshop on Neural Generation and Translation (D19-56)
Copied to clipboard
Hiroaki Hayashi, Yusuke Oda, Alexandra Birch, Ioannis Konstas, Andrew Finch, Minh-Thang Luong, Graham Neubig, Katsuhito Sudoh
| Challenge: | The 3rd Workshop on Neural Machine Translation and Generation (WNGT) was held in concert with the annual conference of the Empirical Methods in Natural Language Processing (EMNLP 2019). |
| Approach: | They describe the results of the third workshop on Neural Generation and Translation held in concert with the annual conference of the Empirical Methods in Natural Language Processing (EMNLP 2019). |
| Outcome: | The results of the 3rd Workshop on Neural Machine Translation and Generation (WNGT) were summarized in Sections 3 and 4. |
Mapping Language to Code in Programmatic Context (D18-1)
Copied to clipboard
| Challenge: | Existing approaches for automatically mapping natural language to executable code have considered limited language or code environments. |
| Approach: | They propose a task of generating class member functions given English documentation and the programmatic context provided by the rest of the class. |
| Outcome: | The proposed model can generate member functions from documentation and the class environment. |
Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent model editing techniques can introduce large unwanted side effects, a new study shows . existing specificity benchmarks do not detect these unwanted side-effects . a recent study shows that model edits can cause significant performance drop . |
| Approach: | They extend existing CounterFact benchmark to include a dynamic component and propose a new benchmark to evaluate model editing techniques. |
| Outcome: | The proposed benchmark improves existing benchmarks for specificity and avoids unwanted side effects. |
An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games (2021.eacl-main)
Copied to clipboard
Alessandro Suglia, Yonatan Bisk, Ioannis Konstas, Antonio Vergari, Emanuele Bastianelli, Andrea Vanzo, Oliver Lemon
| Challenge: | Using guessing games, an artificial agent can learn to perform on novel downstream tasks such as Visual Question Answering (VQA). |
| Approach: | They propose a supervised learning scenario in which an agent learns to mimic successful guessing games and a novel way for an agent to play by itself, called Self-play via Iterated Experience Learning. |
| Outcome: | The proposed model can be applied to a VQA dataset using a supervised learning scenario and a novel way for an agent to play by itself. |