Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations)
TOPICAL: TOPIC Pages AutomagicaLly (2024.naacl-demo)
Copied to clipboard
| Challenge: | Topic pages aggregate useful information about an entity or concept into a single concise article. |
| Approach: | They propose a web app that generates topic pages for biomedical entities on demand . they use large language models and retrieval-augmented generation to generate high-quality topics . |
| Outcome: | The proposed method is based on a human evaluation of 150 biomedical topics . it uses large language models and retrieval-augmented generation (RAG) |
Low-code LLM: Graphical User Interface over Large Language Models (2024.naacl-demo)
Copied to clipboard
Yuzhe Cai, Shaoguang Mao, Wenshan Wu, Zehua Wang, Yaobo Liang, Tao Ge, Chenfei Wu, WangYou WangYou, Ting Song, Yan Xia, Nan Duan, Furu Wei
| Challenge: | Low-code LLM is a visual programming interface that allows users to incorporate their ideas into the process without writing trivial prompts. |
| Approach: | They propose a human-LLM interaction framework that incorporates low-code visual programming interactions to achieve more controllable and stable responses. |
| Outcome: | The proposed framework enables users to incorporate ideas into the process without writing trivial prompts. |
EdTec-QBuilder: A Semantic Retrieval Tool for Assembling Vocational Training Exams in German Language (2024.naacl-demo)
Copied to clipboard
| Challenge: | Existing methods to gather test items from validated item databases are under-researched, but there is little research on assembling exam items from a database of valid items. |
| Approach: | They propose to use semantic search to assist vocational educators in assembling exam forms by using eight retrieval strategies and 25 popular sentence similarity models. |
| Outcome: | The proposed tool is based on eight retrieval strategies and 25 popular pre-trained sentence similarity models. |
DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models (2024.naacl-demo)
Copied to clipboard
| Challenge: | DIALIGHT is a toolkit for developing and evaluating multilingual Task-Oriented Dialogue systems. |
| Approach: | They propose a toolkit for developing and evaluating multilingual Task-Oriented Dialogue systems which facilitates systematic evaluations and comparisons between ToD systems using pretrained language models and those utilising the zero-shot and in-context learning capabilities of Large Language Models. |
| Outcome: | The toolkit enables systematic evaluations between ToD systems using pretrained language models and those utilising the zero-shot and in-context learning capabilities of Large Language Models (LLMs). |
RTSUM: Relation Triple-based Interpretable Summarization with Multi-level Salience Visualization (2024.naacl-demo)
Copied to clipboard
| Challenge: | Abstractive summarization has emerged as a critical tool in the era of information overload. |
| Approach: | They propose an unsupervised summarization framework that utilizes relation triples as the basic unit for summarizing. |
| Outcome: | The proposed framework visualizes salience levels for sentences, relation triples, and phrases. |
Edu-ConvoKit: An Open-Source Library for Education Conversation Data (2024.naacl-demo)
Copied to clipboard
| Challenge: | Edu-ConvoKit is an open-source library for analyzing education conversation data. |
| Approach: | They introduce Edu-ConvoKit, an open-source library for conversation data analysis. |
| Outcome: | The open-source library handles pre-processing, annotation and analysis of education conversation data. |
jp-evalb: Robust Alignment-based PARSEVAL Measures (2024.naacl-demo)
Copied to clipboard
| Challenge: | evalb is used for constituency parsing evaluation, but imposes constraints and requires consistent tokenization and sentence boundary outcomes. |
| Approach: | They propose an evaluation system designed to compute PARSEVAL measures, offering a viable alternative to evalb commonly used for constituency parsing evaluation. |
| Outcome: | The proposed evaluation system is based on an alignment method that aligns sentences and words when discrepancies arise. |
OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMs (2024.naacl-demo)
Copied to clipboard
| Challenge: | Current research seeks to de-bias such models, or suppress potentially biased answers. |
| Approach: | They present a web demo to test the biases of instruction-tuned Large Language Models . they identify 11 different biase based on a corpus of data . |
| Outcome: | The proposed demo shows that biases in instruction-tuning are explicit and transparent . the demo shows how the model was trained and showcases the web application . |
ATLAS: A System for PDF-centric Human Interaction Data Collection (2024.naacl-demo)
Copied to clipboard
| Challenge: | Recent advances in AI only make the importance of high-quality data more pronounced. |
| Approach: | They propose to use the Portable Document Format (PDF) as a data format to better support researchers in collecting rich PDF-centric datasets from users. |
| Outcome: | The proposed toolkit and extensible schema allows researchers to customize the data collection tasks for a variety of purposes, including annotations, drawing, and reading behavior analytics. |
BeLeaf: Belief Prediction as Tree Generation (2024.naacl-demo)
Copied to clipboard
| Challenge: | a novel approach to predicting source-and-target factuality is presented . our linearized tree generation task fully accounts for the factuity tree structure . |
| Approach: | They propose a linearized tree generation task which fully accounts for factuality . they then create a system which leverages the linearized representation to create visualizations . |
| Outcome: | The proposed model and representation fully account for the factuality tree structure, generating the full chain of nested sources instead of the last source only. |
QueryExplorer: An Interactive Query Generation Assistant for Search and Exploration (2024.naacl-demo)
Copied to clipboard
| Challenge: | Formulating effective search queries can be a daunting task for users when they lack expertise in a specific domain or are not proficient in the language of the content. |
| Approach: | QueryExplorer is an interactive query generation, reformulation, and retrieval interface with support for Hug-gingFace generation models and PyTerrier’sretrieval pipelines and datasets. |
| Outcome: | QueryExplorer is an interactive query generation, reformulation, and retrieval interface with support for Hug-gingFace generation models and PyTerrier’sretrieval pipelines and datasets, and extensivelogging of human feedback. |
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2024.naacl-demo)
Copied to clipboard
| Challenge: | Foundation models have demonstrated a great ability to achieve general human-level intelligence far beyond traditional approaches. |
| Approach: | They propose a toolkit to simplify the finetuning of general foundation models. |
| Outcome: | The proposed toolkit simplifies the domain- and task-aware finetuning of general foundation models with limited computing resources. |
DOCMASTER: A Unified Platform for Annotation, Training, & Inference in Document Question-Answering (2024.naacl-demo)
Copied to clipboard
| Challenge: | DOCMASTER is a platform for annotating PDF documents, model training, and inference, tailored to document question-answering. |
| Approach: | They propose to integrate layout information into a unified platform for annotating PDF documents, model training, and inference tailored to document question-answering. |
| Outcome: | The proposed platform is designed for annotating PDF documents, model training, and inference, tailored to document question-answering. |
RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs (2024.naacl-demo)
Copied to clipboard
| Challenge: | Recent advances in machine learning (ML) are attributed to large language models (LLMs), but their escalating memory requirements require developers to partition a large model to distribute it across multiple GPUs or TPUs. |
| Approach: | They propose a lightweight and user-friendly tool to automate distributed training and inference for LLMs and to simplify ML pipeline development. |
| Outcome: | The proposed tool automates distributed training and inference for LLMs, and simplifies ML pipeline development. |
Concept Over Time Analysis: Unveiling Temporal Patterns for Qualitative Data Analysis (2024.naacl-demo)
Copied to clipboard
| Challenge: | Concept Over Time Analysis is a machine-learning-based feature that allows users to define, refine, and visualize concepts of interest within an interactive interface. |
| Approach: | They propose to extend the Discourse Analysis Tool Suite with Concept Over Time Analysis extension that allows users to define, refine, and visualize their concepts of interest within an interactive interface. |
| Outcome: | The proposed system allows users to define, refine, and visualize their concepts of interest within an interactive interface. |
pyvene: A Library for Understanding and Improving PyTorch Models via Interventions (2024.naacl-demo)
Copied to clipboard
Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christopher Manning, Christopher Potts
| Challenge: | Existing libraries are often project-based, but pyvene provides a unified and extensible framework for performing interventions on neural models and sharing the intervened upon models with others. |
| Approach: | They propose an open-source Python library that supports customizable interventions on a range of different PyTorch modules. |
| Outcome: | The proposed framework provides a unified and extensible framework for performing interventions on neural models and sharing the intervened upon models with others. |
Newspaper Signaling for Crisis Prediction (2024.naacl-demo)
Copied to clipboard
| Challenge: | Existing systems for detecting crisis-related signals are limited due to unstructured data, media, and cultural bias, and multiple languages. |
| Approach: | They propose a model for multi-lingual and open-domain newspaper signaling for detecting crisis-related indicators in newspaper articles. |
| Outcome: | The proposed model can detect crisis-related indicators in multiple languages and can be used in open crisis domains in real-time. |
FastFit: Fast and Effective Few-Shot Text Classification with a Multitude of Classes (2024.naacl-demo)
Copied to clipboard
| Challenge: | Few-shot prompting of large language models (LLMs) via API calls presents a unique challenge when dealing with a multitude of classes that share similar semantic meanings. |
| Approach: | They present a Python package that integrates batch contrastive learning and token-level similarity score to provide fast few-shot classification. |
| Outcome: | The proposed method significantly improves multi-class classification speed and accuracy across English and Multilingual datasets. |
AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents (2024.naacl-demo)
Copied to clipboard
Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, Carolin Lawrence
| Challenge: | Existing benchmarks are narrow and simply compute overall task success. |
| Approach: | They propose a framework where both benchmarks and metrics are modular and easily extensible through well documented and easy-to-use APIs. |
| Outcome: | The proposed framework can track agent progress on two use cases and identify common failure points and refine the agent architecture to obtain a significant performance increase. |
ZhuJiu-Knowledge: A Fairer Platform for Evaluating Multiple Knowledge Types in Large Language Models (2024.naacl-demo)
Copied to clipboard
| Challenge: | evaluating the knowledge of large language models (LLMs) is crucial, and rapid advancement in large language modeling has heightened the importance of model evaluations. |
| Approach: | They propose a fairer benchmark for evaluating multiple knowledge types of LLMs by focusing on commonsense knowledge, world knowledge, and language knowledge. |
| Outcome: | The proposed framework evaluates 14 current mainstream LLMs and provides a detailed discussion and analysis of their results. |
Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI (2024.naacl-demo)
Copied to clipboard
Elron Bandel, Yotam Perlitz, Elad Venezian, Roni Friedman, Ofir Arviv, Matan Orbach, Shachar Don-Yehiya, Dafna Sheinwald, Ariel Gera, Leshem Choshen, Michal Shmueli-Scheuer, Yoav Katz
| Challenge: | Textual data processing pipelines are tailored to specific datasets, task and model combinations. |
| Approach: | They propose a library for customizable textual data preparation and evaluation tailored to generative language models. |
| Outcome: | Unitxt is a library for customizable textual data preparation and evaluation tailored to generative language models. |