OpenHermes-Llama-3.2-1B
Model Description
Thanks Redmond.AI for GPU Sponsor!
This model is a fine-tuned version of meta-llama/Llama-3.2-1B on the OpenHermes-2.5 dataset. It is based on the Llama 3.2 architecture, which is an optimized transformer model designed for multilingual dialogue use cases, including agentic retrieval and summarization tasks.
Key Features:
- Base Model: Meta Llama 3.2 1B
- Fine-tuning Dataset: OpenHermes 2.5
- Architecture: Auto-regressive language model with optimized transformer architecture
- Params: 1.23B
- Context Length: 128k
- Input/Output Modalities: Multilingual Text and code
- Supported Languages: Primarily English, with potential for other languages supported by Llama 3.2
Intended Uses & Limitations
This model is intended for commercial and research use in multiple languages, particularly suited for assistant-like chat and agentic applications such as knowledge retrieval, summarization, and query rewriting. It inherits the capabilities and limitations of both the Llama 3.2 1B base model and the OpenHermes 2.5 dataset.
Out of Scope:
- Use that violates applicable laws or regulations
- Use prohibited by the Acceptable Use Policy and Llama 3.2 Community License
- Use in unsupported languages without proper evaluation and safety measures
Training Procedure
Training Data
The model was fine-tuned on the OpenHermes 2.5 dataset, which contains 1M primarily synthetically generated instruction and chat samples. This dataset is a compilation of various open-source datasets and custom-created synthetic datasets, designed to enhance the model's performance in instruction-following and chat scenarios.
Training Results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.1101 | 0.0003 | 1 | 0.9499 |
| 0.7977 | 0.5000 | 1438 | 0.8729 |
| 0.8338 | 1.0000 | 2876 | 0.8647 |
| 0.7714 | 1.4981 | 4314 | 0.8637 |
| 0.8305 | 1.9983 | 5752 | 0.8612 |
| 0.6801 | 2.4963 | 7190 | 0.8631 |
Evaluation Results
The model achieves a final validation loss of 0.8631.
Ethical Considerations and Limitations
Users should be aware of potential biases in the training data and exercise caution when deploying the model, especially in sensitive applications. The model's outputs should be carefully monitored and filtered for inappropriate content.
For more information on the base Llama 3.2 model, please refer to the official Llama 3.2 model card.
- Downloads last month
- 76