Papers by Jana Diesner
StereoMap: Quantifying the Awareness of Human-like Stereotypes in Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been observed to encode harmful associations present in the training data. |
| Approach: | They propose a framework to map LLMs' perceptions of how demographic groups have been viewed by society using the dimensions of Warmth and Competence. |
| Outcome: | The proposed framework maps LLMs’ perceptions of social groups using the dimensions of Warmth and Competence. |
A Constituency Parsing Tree based Method for Relation Extraction from Abstracts of Scholarly Publications (D19-53)
Copied to clipboard
| Challenge: | Existing methods for relation extraction rely on lexical patterns and dependency templates. |
| Approach: | They propose a rule-based method for extracting entity networks from scientific literature . they use syntactic features of constituent parsing trees to extract and construct graphs . |
| Outcome: | The proposed method outperforms state-of-the-art methods in several cases. |
Semantic Networks Extracted from Students’ Think-Aloud Data are Correlated with Students’ Learning Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Largescale open online courses (MOOCs) are available to hundreds of millions of learners, but efficiently evaluating these students' performance remains a crucial task for educators. |
| Approach: | They propose to use textbook-based information as a semantic network to extract concepts and relations from students' verbal data. |
| Outcome: | The proposed models extract concepts and relations from students’ verbal data and show that denser and more interconnected networks were associated with more elaborated knowledge acquisition. |
Tales of Morality: Comparing Human- and LLM-Generated Moral Stories from Visual Cues (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has found that stories are central to how humans communicate moral values . |
| Approach: | They compare human- and LLM-generated moral narratives based on images annotated by humans for moral content . authors propose a framework for evaluating moral storytelling in vision-language models . |
| Outcome: | The proposed model compared human- and LLM-generated narratives on images . human stories reflect a balanced distribution of moral foundations and coherent narrative arcs, but LLMs emphasize Care foundation and lack emotional resolution. |
An Empirical Methodology for Detecting and Prioritizing Needs during Crisis Events (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Social media platforms such as Twitter contain a vast amount of information about the general public’s needs. |
| Approach: | They propose to use Twitter to extract a list of needed resources and detecting sentences that specify who-needs-what resources. |
| Outcome: | The proposed methods achieve 0.64 precision on a set of 1,000 annotated tweets and achieve 0.68 F1-score. |
Detecting Impact Relevant Sections in Scientific Research (2024.lrec-main)
Copied to clipboard
| Challenge: | Impact assessment is an evolving area of research that aims at measuring and predicting the potential effects of projects or programs. |
| Approach: | They propose a framework for automatically assessing the impact of scientific research by identifying pertinent sections in project reports that indicate potential impacts. |
| Outcome: | The proposed method achieves accuracy scores up to 0.81 and is generalizable to scientific research from different domains and languages. |
MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment (2025.findings-acl)
Copied to clipboard
Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, Hinrich Schuetze
| Challenge: | Existing benchmarks for multilinguality for English-centric large language models focus on classic tasks or cover a minimal number of languages. |
| Approach: | They propose a method to assess multilingual capabilities of pre-trained LLMs using parallel sentences. |
| Outcome: | The proposed method evaluates the multilingual capabilities of pre-trained English-centric models using parallel sentences. |
SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have used prompt-based fine-tuning methods for text classification tasks . however, the difficulty and costs of manually selecting domain label terms for the verbalizer remain unexplored . |
| Approach: | They propose a framework to automatically retrieve scientific topic-related terms for low-resource text classification tasks. |
| Outcome: | The proposed method outperforms state-of-the-art methods on scientific text classification tasks under few and zero-shot settings. |
BACO: A Background Knowledge- and Content-Based Framework for Citing Sentence Generation (2021.acl-long)
Copied to clipboard
| Challenge: | citing sentences capture salient information in cited papers and the connection between citing and citing papers. |
| Approach: | They propose a BAckground knowledge- and COntent-based framework for citing sentence generation that integrates two types of information: background knowledge and content. |
| Outcome: | The proposed framework outperforms baselines in the citation sentence generation task. |
CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been widely adopted for a diverse range of tasks, from highly skilled professionals to non-technical individuals. |
| Approach: | They propose a suite of lightweight adversarial attacks that allow LLMs to reveal harmful behavior during conversations. |
| Outcome: | The proposed model can recover from fabricated bias claim and reject biased follow-up questions. |
Detection and Mitigation of the Negative Impact of Dataset Extractivity on Abstractive Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have shown that extractivity can affect output extractivity and the amount of factual information (i.e. faithfulness) in abstractive summarization models. |
| Approach: | They propose to design copy labels to fix the model's copying behaviors and train the model with a copy mechanism to reduce the negative impact of high extractivity on model performance. |
| Outcome: | The proposed method outperforms several competitive baselines and shows that low extractivity can improve model performance, while higher extractivity leads to a tendency for the model to copy text continuously from the source document rather than identifying and summarizing important content. |
TIGEr: Text-to-Image Grounding for Image Caption Evaluation (D19-1)
Copied to clipboard
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, Jianfeng Gao
| Challenge: | Existing metrics based on text-level comparisons fail to assess the quality of captions produced by machines. |
| Approach: | They propose to use a machine-learned text-image grounding model to measure the accuracy of machine-generated captions and their correlation with human judgments. |
| Outcome: | The proposed metric has higher consistency with human judgments and is more accurate than existing metrics. |
Beyond Citations: Corpus-based Methods for Detecting the Impact of Research Outcomes on Society (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for assessing the impact of research are ineffective for identifying impact beyond academia and text-based indicators beyond those that capture attention. |
| Approach: | They propose a deductive and inductive approach to categorize research impact categories using a corpus-based approach . they use a combination of deductive methods and machine learning to infer impact categories from project reports. |
| Outcome: | The proposed method predicts deductively and inductively derived impact categories with 76.39% accuracy and 78.81% accuracy. |
REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning (D19-1)
Copied to clipboard
| Challenge: | Existing metrics for image captioning evaluation provide an overall quality score, which is difficult to infer specific description errors. |
| Approach: | They propose a fine-grained evaluation method REO for automatically measuring the performance of image captioning systems. |
| Outcome: | The proposed method achieves higher consistency with human judgments and provides more intuitive evaluation results than other metrics. |