Buy Tesla, Sell Ford: Assessing Implicit Stock Market Preference in Pre-trained Language Models (2022.acl-short)
Copied to clipboard
| Challenge: | Pretrained language models such as BERT have been used in many NLP tasks . however, there are still significant differences in their implicit preferences in the stock market . |
| Approach: | They assess the implicit stock market preferences in pretrained language models such as BERT . they find that there are significant differences in preferences between industry sectors . |
| Outcome: | The proposed model is more positive towards the stock market, but there are significant differences between industry sectors or within a sector. |
Similar Papers
Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing financial PLMs are not pretrained on sufficiently diverse financial data, leading to subpar generalization performance. |
| Approach: | They propose to pretrain financial PLMs on financial corpus and train financial models on financial data. |
| Outcome: | The proposed financial language models outperform existing financial PLMs on financial tasks even for unseen corpus groups. |
Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that domain-specific BERT models can be improved when in-domain data is used for pretraining. |
| Approach: | They propose to use Twitter and forum text as pretraining sources for two BERT models and use similarity measures to nominate in-domain data for pretraining. |
| Outcome: | The proposed method can be used to improve performance on downstream tasks by using in-domain data. |
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Pretraining by language modeling has become popular but we have yet to understand what language models learn during that process. |
| Approach: | They propose diagnostics that ask questions about information used by language models for generating predictions in context. |
| Outcome: | The proposed diagnostics can be used to study the popular BERT model . they show that the model can distinguish good from bad completions, but struggles with inference and role-based event prediction. |
On the use of BERT for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Existing studies on using pretrained language models for supervised NMT have not been successful. |
| Approach: | They propose to integrate BERT pretrained models with supervised NMT models by using monolingual data. |
| Outcome: | The proposed models improve translation quality in English-German, English-Russian and IWSLT14 datasets. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |
Fin-Bias: Comprehensive Evaluation for LLM Decision-Making under human bias in Finance Domain (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models (LLMs) are limited to small sample and fail to demonstrate LLM susceptibility to context with potential human bias. |
| Approach: | They propose a benchmark for evaluating LLM investment decision-making when faced with uncertainty and possible human-biased opinions. |
| Outcome: | The proposed model can herd the explicit bias in context and even exceed human performance in predicting future stock return. |
BERT Rediscovers the Classical NLP Pipeline (P19-1)
Copied to clipboard
| Challenge: | Pre-trained text encoders have advanced the state of the art on many NLP tasks . Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically . |
| Approach: | They aim to quantify where linguistic information is captured within a network model . they aim to use pre-trained text encoders to displace static word embeddings . |
| Outcome: | The proposed model can adjust the pipeline dynamically, revealing lower-level decisions on the basis of disambiguation from higher-level representations. |
Understanding Pre-trained BERT for Aspect-based Sentiment Analysis (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show impressive results on aspects-based sentiment analysis tasks. |
| Approach: | They analyze the attentions and learned representations of BERT for aspects-based sentiment analysis tasks. |
| Outcome: | The proposed model can be used for aspects-based sentiment analysis (ABSA) but it is not clear how it can provide important features for downstream tasks. |
A Comparison between Pre-training and Large-scale Back-translation for Neural Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | BERT is a promising technique to improve NMT, but how it outperforms standard NMT is understudied. |
| Approach: | We compare MT engines trained with pre-trained BERT and back-translation with incrementally larger amounts of data. |
| Outcome: | The proposed technique outperforms standard NMT models on morphology and syntax. |
ABNIRML: Analyzing the Behavior of Neural IR Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Pretrained contextualized language models such as BERT and T5 have established a new state-of-the-art for ad-hoc ranking. |
| Approach: | They propose a framework for Analyzing the Behavior of Neural IR ModeLs that includes new types of diagnostic probes that allow us to test several characteristics that are not addressed by previous techniques. |
| Outcome: | The proposed framework tests writing styles, factuality, sensitivity to paraphrasing and word order, and can be used to identify unintended biases. |