Mistral-7B-Instruct-v0.2-code-ft

I'm thrilled to introduce the latest iteration of our model, Mistral-7B-Instruct-v0.2-code-ft. This updated version is designed to further enhance coding assistance and co-pilot functionalities. We're eager for developers and enthusiasts to try it out and provide feedback!

Additional Information

This version builds upon the previous Mistral-7B models, incorporating new datasets and features for a more refined experience.

Prompt template: ChatML

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Eval Plus Performance

For detailed performance metrics, visit Eval Plus page: Mistral-7B-Instruct-v0.2-code-ft Eval Plus

Score: 0.421

Dataset:

The model has been trained on a new dataset to improve its performance and versatility:

path: ajibawa-2023/Code-74k-ShareGPT

type: sharegpt

conversation: chatml

Find more about the dataset here: Code-74k-ShareGPT Dataset

Model Architecture

Base Model: mistralai/Mistral-7B-Instruct-v0.2
Tokenizer Type: LlamaTokenizer
Model Type: MistralForCausalLM
Is Mistral Derived Model: true
Sequence Length: 16384 with sample packing

Enhanced Features

Adapter: qlora
Learning Rate: 0.0002 with cosine lr scheduler
Optimizer: adamw_bnb_8bit
Training Enhancements: bf16 training, gradient checkpointing, and flash attention

Download Information

You can download and explore this model through these links on Hugging Face.

Contributions and Feedback

We welcome contributions and feedback from the community. Please feel free to open issues or pull requests on repository.