Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop
AutoAugment Is What You Need: Enhancing Rule-based Augmentation Methods in Low-resource Regimes (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing methods for text data augmentation suffer from potential semantic damage due to the discrete nature of sentences. |
| Approach: | They propose to adapt AutoAugment to solve this problem by using softEDA to increase text data. |
| Outcome: | The proposed method can boost existing augmentation methods and enhance cutting-edge pretrained language models. |
Generating Diverse Translation with Perturbed kNN-MT (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing methods to generate multiple translation candidates do not address the overcorrection problem, which discourages the model from generating synonymous expressions and leans toward gold standards, reducing the diversity in the candidates. |
| Approach: | They propose to introduce perturbed k-nearest neighbor machine translation (kNN-MT) to generate more diverse translations. |
| Outcome: | The proposed methods significantly improve candidate diversity and control diversity by tuning the perturbation’s magnitude. |
The KIND Dataset: A Social Collaboration Approach for Nuanced Dialect Data Collection (2024.eacl-srw)
Copied to clipboard
Asma Yamani, Raghad Alziyady, Reem AlYami, Salma Albelali, Leina Albelali, Jawharah Almulhim, Amjad Alsulami, Motaz Alfarraj, Rabeah Al-Zaidy
| Challenge: | Nuanced dialects are a linguistic variant that pose several challenges for NLP models and techniques. |
| Approach: | They propose an approach to collect high quality Arabic dialect data by social collaboration . they use short texts to collect Arabic dialects and a KIND corpus . |
| Outcome: | The proposed approach is based on the KIND corpus of Arabic dialect data . it provides a high quality dataset and is versatile enough to be multipurpose . |
Can Stanza be Used for Part-of-Speech Tagging Historical Polish? (2024.eacl-srw)
Copied to clipboard
| Challenge: | The performance of Stanza on historical data is evaluated to assess its potential use for automated annotation. |
| Approach: | They evaluate the performance of a part-of-speech (POS) tagger developed for modern Polish on historical texts to assess its potential use for automated annotation. |
| Outcome: | The proposed POS tagger performs well on historical Polish data and pinpoints some common features of misclassified tokens. |
Toward Zero-Shot Instruction Following (2024.eacl-srw)
Copied to clipboard
| Challenge: | a novel approach to zero-shot cross-task generalization is proposed . prior work relied on demonstrations, but this approach could be overestimated . |
| Approach: | They propose a "demonstration-driven instruction following" setting for zero-shot cross-task generalization . they propose to automatically find out critical sentences in a paragraph-style task definition . |
| Outcome: | The proposed approach yields state-of-the-art performance on the Super-NaturalInstructions. |
UnMASKed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts (2024.eacl-srw)
Copied to clipboard
| Challenge: | Language models (LMs) often include societal biases encoded in the human-produced datasets used for their training. |
| Approach: | They evaluated six prominent language models: BERT, RoBERTa, DistilBERT, BERT- multilingual, XLM-RoBERT and DistilberT- multilinguistic. |
| Outcome: | The results show that the models generated by the models were stereotypically gendered and with a reduced bias in multilingual variants. |
Distribution Shifts Are Bottlenecks: Extensive Evaluation for Grounding Language Models to Knowledge Bases (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing benchmarks fail to reflect robustness challenges and fairly evaluate models. |
| Approach: | They propose to ground language models to knowledge bases to investigate distribution shifts in language and linguistic aspects of distribution shift. |
| Outcome: | The proposed method fails to evaluate language models in large and small datasets . the proposed model fails to cope with unseen schemas and language variations . |
AttriSage: Product Attribute Value Extraction Using Graph Neural Networks (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing methods for extracting attribute value from product descriptions are limited in their accuracy. |
| Approach: | They propose a method for extracting product attribute value from product description using graphs and neural networks. |
| Outcome: | The proposed method improves product description attribute value extraction accuracy compared to baseline methods. |
HypoTermQA: Hypothetical Terms Dataset for Benchmarking Hallucination Tendency of LLMs (2024.eacl-srw)
Copied to clipboard
| Challenge: | Hallucinations pose a significant challenge to the reliability and alignment of Large Language Models (LLMs), limiting their widespread acceptance beyond chatbot applications. |
| Approach: | They propose a framework that combines benchmarking LLMs’ hallucination tendencies with efficient hallucinian detection. |
| Outcome: | The proposed framework provides opportunities to test and improve LLMs and can generate benchmarking datasets tailored to specific domains. |
Arabic Synonym BERT-based Adversarial Examples for Text Classification (2024.eacl-srw)
Copied to clipboard
| Challenge: | Often, research studies quantifying the impact of adversarial text attacks have been applied only to models trained in English. |
| Approach: | They propose a word-level study of adversarial text examples in Arabic . they use a synonym attack with a BERT model to assess their robustness . |
| Outcome: | The proposed attack compares Arabic adversarial examples with their original examples and regains 2% accuracy after training. |
A Hypothesis-Driven Framework for the Analysis of Self-Rationalising Models (2024.eacl-srw)
Copied to clipboard
| Challenge: | Recent advances in LLMs generating longer coherent text have popularised self-rationalising models, which produce a natural language explanation alongside their output. |
| Approach: | They propose a Bayesian network-based hypothesis-driven statistical framework that allows us to judge how similar LLM-generated free-text explanations are to LLMs. |
| Outcome: | The proposed framework does not exhibit a strong similarity to GPT-3.5. |
Align before Attend: Aligning Visual and Textual Features for Multimodal Hateful Content Detection (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to multimodal hateful content detection focus on detecting hate speech from text-based content, but they fail to address modality-specific features. |
| Approach: | They propose a context-aware attention framework for multimodal hateful content detection that integrates an attention layer to meaningfully align the visual and textual features. |
| Outcome: | The proposed framework achieves F1-scores of 69.7% and 70.3% on two hateful meme datasets and shows 2.5% and 3.2% performance improvement over the state-of-the-art systems. |
Topic-guided Example Selection for Domain Adaptation in LLM-based Machine Translation (2024.eacl-srw)
Copied to clipboard
| Challenge: | Current machine translation (MT) systems perform well in domains on which they were trained, but adaptation to unseen domains remains a challenge. |
| Approach: | They propose to use large language models to adapt to unseen domains by in-context example selection. |
| Outcome: | The proposed method outperforms baselines on multilingual out-of-domain tests, though it does not match performance with strong baselines for the in-language setting. |
Reforging : A Method for Constructing a Linguistically Valid Japanese CCG Treebank (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing treebanks for Combinatory Categorial Grammar (CCG) are insufficient for linguistic validity of CCG . |
| Approach: | They propose to combine ABCTreebank and lightblue to generate a linguistically valid Japanese CCG treebank with detailed information by filtering lightblu's lexical items using ABCTtreebank. |
| Outcome: | The proposed method generates a linguistically valid Japanese CCG treebank with detailed information by combining the strengths of ABCTreebank and lightblue. |
Thesis Proposal: Detecting Agency Attribution (2024.eacl-srw)
Copied to clipboard
| Challenge: | 'agency' is the freedom and capacity of an entity to act, and the corresponding Natural Language Processing (NLP) task involves automatically detecting attributions of agency to entities in text. |
| Approach: | They propose a schema to annotate a dataset for agency attribution and formulate additional research questions by applying NLP models. |
| Outcome: | The proposed framework draws on semantic frame analysis, role labelling and related techniques. |
A Thesis Proposal ClaimInspector Framework: A Hybrid Approach to Data Annotation using Fact-Checked Claims and LLMs (2024.eacl-srw)
Copied to clipboard
| Challenge: | Current efforts focus on textual claims sourced mainly from Twitter . lack of automated control measures and reliance on human annotation increase noise risk . |
| Approach: | They propose to use a framework to integrate data annotation to mitigate misinformation . they propose to include fact-checks alongside the corresponding claims made by politicians . |
| Outcome: | The proposed dataset will include fact-checks alongside the corresponding claims made by politicians. |
Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024.eacl-srw)
Copied to clipboard
| Challenge: | a survey examines the landscape of mathematical problem-solving techniques . large language models have proven to be potent assets in unraveling nuances of mathematical reasoning . |
| Approach: | They examine the evolution of Large Language Models (LLMs) for solving mathematical problems . they examine the spectrum of LLM-oriented techniques proposed for solving math problems - and their challenges . |
| Outcome: | The survey examines the spectrum of proposed LLM-oriented techniques in solving math problems. |
Representation and Generation of Machine Learning Test Functions (2024.eacl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been adopted for ML code generation but their implications are relatively unexplored. |
| Approach: | They examine the use of Large Language Models to extract representations of ML source code and tests to understand the semantic relationships between human-written tests and LLM-generated ones. |
| Outcome: | The proposed models can be used to extract representations of ML source code and tests and annotate them for usefulness, documentation, and correctness. |
The Generative AI Paradox in Evaluation: “What It Can Solve, It May Not Evaluate” (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing studies on using Large Language Models for model evaluation have focused on using LLMs for reference-free evaluation to meet the needs of long-form text evaluation. |
| Approach: | They propose to use Large Language Models (LLMs) for generation tasks to evaluate models. |
| Outcome: | The proposed model evaluations show that LLMs are less faithful to evaluation tasks than open-source models. |
Generative Data Augmentation using LLMs improves Distributional Robustness in Question Answering (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing domain adaptation methods do not account for unseen natural distribution shifts. |
| Approach: | They perform experiments on 4 different datasets under varying amounts of distribution shift . they analyze how "in-the-wild" generation can help achieve domain generalization . |
| Outcome: | The proposed approach augments reading comprehension datasets with generated data to improve robustness towards natural distribution shifts. |
Japanese-English Sentence Translation Exercises Dataset for Automatic Grading (2024.eacl-srw)
Copied to clipboard
| Challenge: | Sentence translation exercises (STEs) are used in the early stages of L2 language learning. |
| Approach: | They propose to formalize the task of automatic assessment of Sentence Translation Exercises (STEs) they use a dataset of 21 questions and 3,498 student responses to test the framework . |
| Outcome: | The proposed task is based on a dataset of 21,498 student responses between Japanese and English and compares them to baseline models with fine-tuned BERT and GPT-3.5 with few-shot learning. |
The Impact of Integration Step on Integrated Gradients (2024.eacl-srw)
Copied to clipboard
| Challenge: | Integrated gradients (IG) are a powerful tool for explaining the internal structure of a language model. |
| Approach: | They propose to customize the step count for each instance to minimize the error. |
| Outcome: | The optimal number of steps to maintain minimal error varies from instance to instance. |
GesNavi: Gesture-guided Outdoor Vision-and-Language Navigation (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing datasets for outdoor Vision-and-Language Navigation (VLN) tasks do not include verbal instructions for communicating with mobility. |
| Approach: | They propose a dataset for gesture-guided outdoor VLN instructions with demonstrative expressions that incorporates gestures and linguistic commands. |
| Outcome: | The proposed datasets are compared against existing datasets and analysed in detail. |
Can docstring reformulation with an LLM improve code generation? (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing approaches focus on training, fine-tuning or prompting LLMs to generate better outputs given the same input. |
| Approach: | They propose to optimize part of the input, the docstring, via reformulation with an LLM to improve code generation. |
| Outcome: | The proposed methods improve code generation on the original HumanEval benchmark and multiple curated variants on the same input. |
Benchmarking Diffusion Models for Machine Translation (2024.eacl-srw)
Copied to clipboard
| Challenge: | Diffusion models have shown great potential on many generative tasks, but their application to natural language processing (NLP) is still a less explored direction. |
| Approach: | They adapt two diffusion-based text generation models, Diffusion-LM and DiffuSeq, to perform machine translation. |
| Outcome: | The proposed models struggle more on long-range dependencies than other models. |
Forged-GAN-BERT: Authorship Attribution for LLM-Generated Forged Novels (2024.eacl-srw)
Copied to clipboard
| Challenge: | generative Large Language Models (LLMs) are capable of producing human-like texts, but they pose challenges related to the authenticity of the text documents. |
| Approach: | They propose a modified GANBERT-based model to improve the classification of forged novels via the Forged Novels Generator and the generator in GAN. |
| Outcome: | The proposed model improves classification of forged novels in two data-augmentation aspects. |
Thesis Proposal: Detecting Empathy Using Multimodal Language Model (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing studies on empathy detection in video and audio have relied on scripted or semi-scripted interactions that fail to capture the complexities and nuances of real-life interactions. |
| Approach: | They propose to develop a multimodal language model that detects empathy in audiovisual data by using neural architecture search and optimisation techniques. |
| Outcome: | The proposed model will be able to detect empathy in audiovisual data and use neural architecture search to deliver it. |
Toward Sentiment Aware Semantic Change Analysis (2024.eacl-srw)
Copied to clipboard
| Challenge: | Current approaches to analyze semantic change are lagging behind . current methods only detect semantic change as a binary classification or graded change scores . |
| Approach: | They propose to augment models of semantic change with sentiment information . they demonstrate that existing models extract reliable sentiment information from historical corpora . |
| Outcome: | The proposed approach shows mixed results on the English SemEval of Lexical Semantic Change and its associated historical corpora. |
Dynamic Task-Oriented Dialogue: A Comparative Study of Llama-2 and Bert in Slot Value Generation (2024.eacl-srw)
Copied to clipboard
| Challenge: | Recent advances in instruction-based language models have demonstrated exceptional performance across various natural language processing tasks. |
| Approach: | They propose to use BERT and Llama-2 to generate dynamic task-oriented dialogues using a multi-dimensional dataset. |
| Outcome: | The proposed models generate predictions for masked slot values within text and are reproducible in open-source environments. |