Papers by Hendrik Strobelt
GLTR: Statistical Detection and Visualization of Generated Text (P19-3)
Copied to clipboard
| Challenge: | GLTR is a tool to detect generated text that can be used by non-experts. |
| Approach: | They propose a tool to detect generated text using a set of statistical methods that can be used by non-experts. |
| Outcome: | The proposed method improves detection rate of fake text from 54% to 72% without training. |
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models (2020.acl-demos)
Copied to clipboard
| Challenge: | Large Transformer-based language models can route and reshape complex information via their multi-headed attention mechanism. |
| Approach: | They propose a tool to help humans conduct flexible, interactive investigations and formulate hypotheses for the model-internal reasoning process. |
| Outcome: | Using exBERT, we can analyze the representations and attentions of large language models and extend them to previously not analyzed models. |
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)
Copied to clipboard
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
| Challenge: | Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. |
| Approach: | They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations. |
| Outcome: | The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work. |
LMdiff: A Visual Diff Tool to Compare Language Models (2021.emnlp-demo)
Copied to clipboard
| Challenge: | LMdiff visually compares probability distributions of two different language models . notably absent from the range of available tools are those that aim to compare distributions produced by different models. |
| Approach: | They propose a tool that visually compares probability distributions of two different language models that differ through finetuning, distillation, or simply training with different parameter sizes. |
| Outcome: | The proposed tool allows the generation of hypotheses about model behavior by investigating text instances token by token and further assists in choosing interesting text instances from large corpora. |
Multi-Level Explanations for Generative Language Models (2025.acl-long)
Copied to clipboard
Lucas Monteiro Paes, Dennis Wei, Hyo Jin Do, Hendrik Strobelt, Ronny Luss, Amit Dhurandhar, Manish Nagireddy, Karthikeyan Natesan Ramamurthy, Prasanna Sattigeri, Werner Geyer, Soumya Ghosh
| Challenge: | Large language models (LLMs) are being used for context-grounded tasks like summarizing meetings and answering doctors' questions. |
| Approach: | They propose a technique to provide explanations for context-grounded text generation by assigning scores to parts of the context to quantify their influence on the model output. |
| Outcome: | The proposed framework can provide more faithful explanations of generated output than available alternatives, including LLM self-explanations. |