Edit model card

CryptoBERT

This model is a fine-tuned version of ProsusAI/finbert on the Custom Crypto Market Sentiment dataset. It achieves the following results on the evaluation set:

  • Loss: 0.3823
from transformers import BertTokenizer, BertForSequenceClassification
from transformers import pipeline

tokenizer = BertTokenizer.from_pretrained("kk08/CryptoBERT")
model = BertForSequenceClassification.from_pretrained("kk08/CryptoBERT")

classifier = pipeline("sentiment-analysis", model=model, tokenizer=tokenizer)
text = "Bitcoin (BTC) touches $29k, Ethereum (ETH) Set To Explode, RenQ Finance (RENQ) Crosses Massive Milestone"
result = classifier(text)
print(result)
[{'label': 'LABEL_1', 'score': 0.9678454399108887}]

Model description

This model fine-tunes the ProsusAI/finbert, which is a pre-trained NLP model to analyze the sentiment of the financial text. CryptoBERT model fine-tunes this by training the model as a downstream task on Custom Crypto Sentiment data to predict whether the given text related to the Crypto market is Positive (LABEL_1) or Negative (LABEL_0).

Intended uses & limitations

The model can perform well on Crypto-related data. The main limitation is that the fine-tuning was done using only a small corpus of data

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 16
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 10

Training results

Training Loss Epoch Step Validation Loss
0.4077 1.0 27 0.4257
0.2048 2.0 54 0.2479
0.0725 3.0 81 0.3068
0.0028 4.0 108 0.4120
0.0014 5.0 135 0.3566
0.0007 6.0 162 0.3495
0.0006 7.0 189 0.3645
0.0005 8.0 216 0.3754
0.0004 9.0 243 0.3804
0.0004 10.0 270 0.3823

Framework versions

  • Transformers 4.28.0
  • Pytorch 2.0.0+cu118
  • Datasets 2.11.0
  • Tokenizers 0.13.3
Downloads last month
1,665
Safetensors
Model size
109M params
Tensor type
I64
ยท
F32
ยท
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for kk08/CryptoBERT

Base model

ProsusAI/finbert
Finetuned
(24)
this model

Spaces using kk08/CryptoBERT 3