Papers by Scott Martin
The TechQA Dataset (2020.acl-main)
Copied to clipboard
Vittorio Castelli, Rishav Chakravarti, Saswati Dana, Anthony Ferritto, Radu Florian, Martin Franz, Dinesh Garg, Dinesh Khandelwal, Scott McCarley, Michael McCawley, Mohamed Nasr, Lin Pan, Cezar Pendus, John Pitrelli, Saurabh Pujar, Salim Roukos, Andrzej Sakrajda, Avi Sil, Rosario Uceda-Sosa, Todd Ward, Rong Zhang
| Challenge: | TECHQA is a domain-adaptation question answering dataset for the technical support domain. |
| Approach: | They propose a domain-adaptation question-answering dataset for the technical support domain that contains actual questions posed by users on a technical forum . |
| Outcome: | The TECHQA dataset highlights two real-world issues from the automated customer support domain. |
MuDoCo: Corpus for Multidomain Coreference Resolution and Referring Expression Generation (2020.lrec-1)
Copied to clipboard
| Challenge: | a new dataset, MuDoCo, is composed of authored dialogs between a fictional user and a system . the dialogs cross domains and users exhibit complex task switching behavior . |
| Approach: | They propose a new dataset, MuDoCo, composed of authored dialogs between a fictional user and a system . they propose two baseline models for the downstream tasks: coreference resolution and referring expression generation. |
| Outcome: | The proposed dataset contains 8,429 dialogs with an average of 5.36 turns per dialog . the users exhibit complex task switching behavior such as re-initiating a previous task . |
PrimeQA: The Prime Repository for State-of-the-Art Multilingual Question Answering Research and Development (2023.acl-demo)
Copied to clipboard
Avi Sil, Jaydeep Sen, Bhavani Iyer, Martin Franz, Kshitij Fadnis, Mihaela Bornea, Sara Rosenthal, Scott McCarley, Rong Zhang, Vishwajeet Kumar, Yulong Li, Md Arafat Sultan, Riyaz Bhat, Juergen Bross, Radu Florian, Salim Roukos
| Challenge: | Question Answering (QA) is a major area of research in Natural Language Processing (NLP) |
| Approach: | They propose a one-stop and open-source QA repository for question answering . it supports core QA functionalities like retrieval and reading comprehension . they say it will facilitate easy replication of state-of-the-art (SOTA) QA methods . |
| Outcome: | The proposed framework enables easy replication of state-of-the-art (SOTA) QA methods. |
Discovering Language Model Behaviors with Model-Written Evaluations (2023.findings-acl)
Copied to clipboard
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
| Challenge: | Prior work creates evaluations with crowdwork or existing data sources, which are not always available. |
| Approach: | They generate evaluations automatically with language models (LMs) using crowdwork or existing data sources to find out how they behave . |
| Outcome: | The results show that large LMs repeat back a dialog user’s preferred answer and express greater desire to pursue concerning goals like resource acquisition and goal preservation. |