A Computational Method for Measuring Open Codes in Qualitative Analysis (2026.findings-acl)
Copied to clipboard
John Chen, Alexandros Nikolaos Lotsos, Sihan Cheng, Lexie Zhao, Yanjia Zhang, Jessica Hullman, Bruce Sherin, Uri Wilensky, Michael Horn
| Challenge: | Qualitative analysis is widely adopted across many social science disciplines. |
| Approach: | They propose a theory-informed computational method for measuring inductive coding results from humans and GAI. |
| Outcome: | The proposed method captures breadth, consensus, unique contribution, and systematic deviation without assuming ground truth. |
Similar Papers
Text Annotation via Inductive Coding: Comparing Human Experts to LLMs in Qualitative Data Analysis (2025.findings-naacl)
Copied to clipboard
| Challenge: | Qualitative data analysis (QDA) is an important research method across fields such as marketing, media studies, social science, psychology, medical research, and others. |
| Approach: | They evaluate the performance of open-source LLMs by comparing them to human experts. |
| Outcome: | The proposed method is based on inductive coding using large language models. |
HICode: Hierarchical Inductive Coding with LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-grained corpus analysis are limited to small subsets and require manual labeling and statistical tools like topic modeling. |
| Approach: | They propose a pipeline that inductively generates labels from analysis data and then hierarchically clusters them to surface emergent themes. |
| Outcome: | The proposed pipeline validates the approach across three datasets and shows it is robust through automated and human evaluations. |
Qualitative Code Suggestion: A Human-Centric Approach to Qualitative Coding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Qualitative coding is a content analysis method that assigns descriptive labels or qualitative codes to passages. |
| Approach: | They propose a qualitative code suggestion task where a ranked list of previously assigned qualitative codes is suggested from an identified passage. |
| Outcome: | The proposed method integrates previously ignored properties such as the sequence in which passages are annotated, the importance of rare codes and the differences in annotation styles between coders. |
Automating Qualitative Data Analysis with Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for qualitative data analysis are far from resembling a human's analysis outcome. |
| Approach: | They propose a method based on Large Language Models to tackle automated coding and make it as close as possible to the results of human researchers. |
| Outcome: | The proposed method is based on large language models and can be as close as possible to the results of human researchers. |
Measuring What Matters: Evaluating Ensemble LLMs with Label Refinement in Inductive Coding (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are prone to inconsistencies and individual biases, limiting their reliability. |
| Approach: | They propose a framework that combines ensemble methods with code refinement methodology to address these challenges. |
| Outcome: | The proposed framework outperforms large language models and LLMs with a low-rank averaging and a moderator-based mechanism to simulate human consensus. |
Can LLMs Understand the Impact of Trauma? Costs and Benefits of LLMs Coding the Interviews of Firearm Violence Survivors (2026.findings-acl)
Copied to clipboard
| Challenge: | Firearm violence research remains underfunded and difficult to scale due to the lack of funding from the NIH and CDC. |
| Approach: | They use open-source large language models to inductively code interviews with 21 Black men who have survived community firearm violence. |
| Outcome: | The use of open-source LLMs to inductively code interviews with 21 Black men shows that the models can identify important codes, but that they are highly sensitive to data processing. |
A Survey of Computational Framing Analysis Approaches (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing computational methods for framing analysis are limited . a lack of a comprehensive understanding of framability is limiting the research . |
| Approach: | They propose to combine existing approaches to analyze large-scale datasets using computational methods. |
| Outcome: | The proposed methods will help scholars better understand how frames are being explored computationally, the authors argue . |
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models (2025.acl-long)
Copied to clipboard
Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Xianzhen Luo, Qiufeng Wang, YuanTao Fan, Qingfu Zhu, Zhaoxiang Zhang, Yang Gao, Jie Fu, Qian Liu, Houyi Li, Ge Zhang, Yuan Qi, Xu Yinghui, Wei Chu, Zili Wang
| Challenge: | Code LLMs lack reproducible data pipelines and training protocols for reproducible advancements in code intelligence. |
| Approach: | They propose a top-tier code LLM that releases model weights and inference code . reproducible data pipelines, rigorous experimental ablation results and training protocols are included . |
| Outcome: | The proposed model achieves comparable performance to leading models and serves as an "open cookbook" reproducible training data, rigorous experimental ablation results, and detailed training protocols are also included in the model. |
CodeArena: Evaluating and Aligning CodeLLMs on Human Preference (2025.emnlp-main)
Copied to clipboard
Jian Yang, Jiaxi Yang, Wei Zhang, Jin Ke, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Zhoujun Li, Binyuan Hui, Junyang Lin
| Challenge: | Code large language models (codeLLMs) focus on synthesizing the correct code snippet, ignoring the alignment with human preferences. |
| Approach: | They propose a benchmark code-based on 40 categories and 44 programming languages to emulate real-world coding tasks. |
| Outcome: | The proposed benchmarks show that open-source code LLMs perform better than open-sourced ones. |
Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant (2026.acl-industry)
Copied to clipboard
Joseph Matveyenko, James Liu, John David Parsons, Ryan Brown, Alina I. Palimaru, Vipul Gupta, Prateek Puri
| Challenge: | Qualitative research emphasizes constructing meaning through iterative engagement with textual data. |
| Approach: | They present and benchmark a qualitative research assistant system that allows researchers to identify themes and annotate datasets. |
| Outcome: | The proposed system achieves an inter-rater reliability between Muse and humans of Cohen’s = 0.7 for well-specified codes. |