README.md

metadata

license: cc-by-nc-4.0
base_model: bigcode/starcoder2-7b
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - code
  - starcoder
  - bigcode
  - sft
  - 7b

Starcoder-2-chat

Starcoder-2-chat is an instruction fine-tuned of bigcode/starcoder2-7b using the glaiveai/glaive-code-assistant-v2 dataset using LoRA.

🏆 Evaluation results

Thanks to Muhammad Bin Usman for running evals on Starcoder2-chat.

HUMANEVAL

0.3231707317073171

HUMANEVALPLUS

0.25609756097560976

INSTRUCT-HUMANEVAL

0.3231707317073171

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-7
• train_batch_size: 2
• eval_batch_size: Not specified
• seed: Not specified
• gradient_accumulation_steps: 8
• total_train_batch_size: Not specified
• optimizer: PagedAdamW with 32-bit precision
• lr_scheduler_type: Cosine
• lr_scheduler_warmup_steps: 100
• training_epoch: 1

Framework versions

• Transformers 4.39.0.dev0
• Peft 0.9.1.dev0
• Datasets 2.18.0
• torch 2.2.0
• accelerate 0.27.2