|
--- |
|
license: mit |
|
language: |
|
- en |
|
datasets: |
|
- TRAC-MTRY/traclm-v3-data |
|
- Open-Orca/SlimOrca-Dedup |
|
--- |
|
# Model Card for traclm-v3-7b-instruct |
|
|
|
An Army domain finetune of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) created by finetuning on a merger of domain-specific and general-purpose instruction-tuning datasets. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
This model is a research project aimed at exploring whether a pretrained LLM can acquire tangible domain-specific knowledge about the Army domain. |
|
|
|
- **Developed by:** The Research and Analysis Center, Army Futures Command, U.S. Army |
|
- **License:** MIT |
|
- **Model Type:** MistralForCausalLM |
|
- **Finetuned from model:** [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) |
|
|
|
### Available Quantizations (for running on low-resource hardware): |
|
- [AWQ](https://huggingface.co/TRAC-MTRY/traclm-v3-7b-instruct-AWQ) |
|
- [GPTQ](https://huggingface.co/TRAC-MTRY/traclm-v3-7b-instruct-GPTQ) |
|
- [GGUF](https://huggingface.co/TRAC-MTRY/traclm-v3-7b-instruct-GGUF) |
|
|
|
### Downstream Use |
|
|
|
This model is instruction-tuned, and is thus more capable of following user instructions than its corresponding base version. However, this model is still capable of extreme hallucination, so all outputs should be verified by end users. |
|
|
|
### Out-of-Scope Use |
|
|
|
The creation of this model constitutes academic research in partnership with the Naval Postgraduate School. The purpose of this research is to inform future DoD experimentation regarding the development and application of domain-specific large language models. Experiments involving direct application of this model to downstream military tasks are encouraged, but extreme caution should be exercised before productionalization. |
|
|
|
## Prompt Format |
|
|
|
This model was fine-tuned with the chatml prompt format. It is *highly* recommended that you use the same format for any interactions with the model. Failure to do so will degrade performance significantly. |
|
|
|
ChatML Format: |
|
``` |
|
<|im_start|>system |
|
Provide some context and/or instructions to the model. |
|
<|im_end|> |
|
<|im_start|>user |
|
The user’s message goes here |
|
<|im_end|> |
|
<|im_start|>assistant |
|
``` |
|
|
|
The ChatML format can easily be applied to text you plan to process with the model using the `chat_template` included in the tokenizer. Read [here](https://huggingface.co/docs/transformers/main/en/chat_templating) for additional information. |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
This model was trained on a shuffled merger of the following datasets: |
|
- General Purpose Instruction Tuning: [Open-Orca/SlimOrca-Dedup](https://huggingface.co/datasets/Open-Orca/SlimOrca-Dedup) |
|
- Domain Specific Instruction Tuning: [TRAC-MTRY/traclm-v3-data](https://huggingface.co/datasets/TRAC-MTRY/traclm-v3-data) **(TBP)** |
|
|
|
### Training Procedure |
|
|
|
The model was trained using Open Access AI Collective's [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) framework and Microsoft's [DeepSpeed](https://github.com/microsoft/DeepSpeed) framework for model/data parallelism. |
|
|
|
### Training Hardware |
|
|
|
Training was conducted on a single compute node 4x NVIDIA A100 GPUs. |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-06 |
|
- train_batch_size: 4 |
|
- eval_batch_size: 4 |
|
- seed: 28 |
|
- distributed_type: multi-GPU |
|
- num_devices: 4 |
|
- total_train_batch_size: 16 |
|
- total_eval_batch_size: 16 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: cosine |
|
- lr_scheduler_warmup_steps: 19 |
|
- num_epochs: 3 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | |
|
|:-------------:|:-----:|:----:|:---------------:| |
|
| 1.5726 | 0.0 | 1 | 1.6102 | |
|
| 1.2333 | 0.2 | 510 | 1.2477 | |
|
| 1.1485 | 0.4 | 1020 | 1.2010 | |
|
| 1.106 | 0.6 | 1530 | 1.1687 | |
|
| 1.1772 | 0.8 | 2040 | 1.1419 | |
|
| 1.1567 | 1.0 | 2550 | 1.1190 | |
|
| 1.0359 | 1.19 | 3060 | 1.1130 | |
|
| 0.945 | 1.39 | 3570 | 1.0977 | |
|
| 0.9365 | 1.59 | 4080 | 1.0831 | |
|
| 0.9334 | 1.79 | 4590 | 1.0721 | |
|
| 0.8913 | 1.99 | 5100 | 1.0627 | |
|
| 0.804 | 2.18 | 5610 | 1.0922 | |
|
| 0.7892 | 2.38 | 6120 | 1.0888 | |
|
| 0.7757 | 2.58 | 6630 | 1.0873 | |
|
| 0.7797 | 2.78 | 7140 | 1.0864 | |
|
|
|
### Framework versions |
|
|
|
- Transformers 4.38.2 |
|
- Pytorch 2.1.2+cu121 |
|
- Datasets 2.18.0 |
|
- Tokenizers 0.15.0 |
|
|
|
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl) |
|
|
|
## Model Card Contact |
|
|
|
MAJ Daniel C. Ruiz (daniel.ruiz@nps.edu) |
