Papers by Joseph James
On the Rigour of Scientific Writing: Criteria, Analysis, and Insights (2024.findings-emnlp)
Copied to clipboard
| Challenge: | despite its importance, little work exists on modelling rigour in scientific writing . despite widespread use of term, scientific literature lacks definition of rigor . |
| Approach: | They propose a framework to automatically identify and define rigour criteria and assess their relevance in scientific writing. |
| Outcome: | The proposed framework can be tailored to the evaluation of scientific rigour for different areas. |
RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluation (2026.findings-acl)
Copied to clipboard
| Challenge: | Scientific rigour tends to be sidelined in favour of bold statements, leading authors to overstate claims beyond what their results support. |
| Approach: | They propose a multimodal framework that retrieves supporting evidence from a paper and assigns each claim an overstatement score. |
| Outcome: | The proposed framework retrieves supporting evidence from ICLR and NeurIPS papers and assigns each claim an overstatement score. |
OctoTools: A Multi-Agent Framework with Extensible Tools for Complex Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing prompting methods for large language models (LLMs) are restricted to specialized domains, limited tool types, or require additional training data. |
| Approach: | They propose a training-free, user-friendly, and easily extensible multi-agent framework designed to tackle complex reasoning across diverse domains. |
| Outcome: | The proposed framework outperforms AutoGen, GPT-Functions, and LangChain by up to 10.6% when given the same set of tools. |
Development and Benchmarking of a Blended Human-AI Qualitative Research Assistant (2026.acl-industry)
Copied to clipboard
Joseph Matveyenko, James Liu, John David Parsons, Ryan Brown, Alina I. Palimaru, Vipul Gupta, Prateek Puri
| Challenge: | Qualitative research emphasizes constructing meaning through iterative engagement with textual data. |
| Approach: | They present and benchmark a qualitative research assistant system that allows researchers to identify themes and annotate datasets. |
| Outcome: | The proposed system achieves an inter-rater reliability between Muse and humans of Cohen’s = 0.7 for well-specified codes. |
Discovering Language Model Behaviors with Model-Written Evaluations (2023.findings-acl)
Copied to clipboard
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
| Challenge: | Prior work creates evaluations with crowdwork or existing data sources, which are not always available. |
| Approach: | They generate evaluations automatically with language models (LMs) using crowdwork or existing data sources to find out how they behave . |
| Outcome: | The results show that large LMs repeat back a dialog user’s preferred answer and express greater desire to pursue concerning goals like resource acquisition and goal preservation. |
LAMP-MedQA: A Lightweight Multi-Agent System for Patient-Oriented Medical Question Answering (2026.acl-srw)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a promising way to bridge the gap between patient health literacy and access to care. |
| Approach: | They evaluate a range of open- and closed-source LLMs on a MeDiSumQA dataset . they propose a lightweight multi-agent framework for patient-oriented medical question answering . |
| Outcome: | The proposed model achieves lower FKGL than zero-shot GPT-5 and highest simplification quality among all models. |