README.md

---
language: 
- ru
- en
tags:
- ruGPT
- GGML
- NLP
- Text Generation
license: "mit"
---

# ruGPT-3.5-13B Converted to GGML Format / ruGPT-3.5-13B Конвертированная в формат GGML

## Model Description / Описание модели

### English

This repository contains a GGML-formatted version of the [ruGPT-3.5-13B model](https://huggingface.co/ai-forever/ruGPT-3.5-13B) originally hosted on Hugging Face. The model has 13 billion parameters and was initially trained on a 300GB dataset from various domains. It was further fine-tuned on 100GB of code and legal documents. The model understands both Russian and English.

#### Dataset Details

- **Training Data**: 300GB from various domains
- **Fine-tuning Data**: 100GB of code and legal documents
- **Technical Specs**: Trained using Deepspeed and Megatron libraries on 300B tokens dataset for 3 epochs, around 45 days on 512 V100 GPUs. Fine-tuned for 1 epoch with a sequence length of 2048, around 20 days on 200 A100 GPUs.
- **Perplexity**: Around 8.8 for Russian language

#### Usage

##### For QuantizationType.Q4_0 and ContainerType.GGJT
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_0-ggjt.bin")
print(model.generate("The meaning of life is ").text)
```

##### For QuantizationType.Q4_0 and ContainerType.GGML
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_0.bin")
print(model.generate("The meaning of life is ").text)
```

##### For QuantizationType.Q4_1 and ContainerType.GGJT
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_1-ggjt.bin")
print(model.generate("The meaning of life is ").text)
```

##### For QuantizationType.Q4_1 and ContainerType.GGML
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_1.bin")
print(model.generate("The meaning of life is ").text)
```

##### For QuantizationType.Q5_0 and ContainerType.GGJT
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_0-ggjt.bin")
print(model.generate("The meaning of life is ").text)
```

##### For QuantizationType.Q5_0 and ContainerType.GGML
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_0.bin")
print(model.generate("The meaning of life is ").text)
```

##### For QuantizationType.Q5_1 and ContainerType.GGJT
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_1-ggjt.bin")
print(model.generate("The meaning of life is ").text)
```

##### For QuantizationType.Q5_1 and ContainerType.GGML
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_1.bin")
print(model.generate("The meaning of life is ").text)
```

##### For QuantizationType.Q8_0 and ContainerType.GGJT
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q8_0-ggjt.bin")
print(model.generate("The meaning of life is ").text)
```

##### For QuantizationType.Q8_0 and ContainerType.GGML
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q8_0.bin")
print(model.generate("The meaning of life is ").text)
```

##### f16 Version

```python
# f16 Version
from llm_rs import AutoModel

model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-f16.bin")
print(model.generate("Смысл жизни в ").text)
```

#### Compatibility

While this model is intended to be compatible with any GGML-compatible UI, it has not been extensively tested in such environments. Use at your own risk.

### Русский

Этот репозиторий содержит версию модели [ruGPT-3.5-13B](https://huggingface.co/ai-forever/ruGPT-3.5-13B) в формате GGML. Модель имеет 13 миллиардов параметров и изначально обучалась на 300ГБ данных из различных доменов. Далее она была дообучена на 100ГБ кода и юридических документов. Модель понимает как русский, так и английский языки.

#### Детали набора данных

- **Тренировочные данные**: 300ГБ из различных доменов
- **Данные для дообучения**: 100ГБ кода и юридических документов
- **Технические характеристики**: Обучена с использованием библиотек Deepspeed и Megatron на наборе данных из 300 миллиардов токенов за 3 эпохи, примерно 45 дней на 512 GPU V100. Дообучена 1 эпоху с длиной последовательности 2048, примерно 20 дней на 200 GPU A100.
- **Перплексия**: Около 8,8 для русского языка

#### Использование

##### Для QuantizationType.Q4_0 и ContainerType.GGJT
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_0-ggjt.bin")
print(model.generate("Смысл жизни в ").text)
```

##### Для QuantizationType.Q4_0 и ContainerType.GGML
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_0.bin")
print(model.generate("Смысл жизни в ").text)
```

##### Для QuantizationType.Q4_1 и ContainerType.GGJT
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_1-ggjt.bin")
print(model.generate("Смысл жизни в ").text)
```

##### Для QuantizationType.Q4_1 и ContainerType.GGML
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q4_1.bin")
print(model.generate("Смысл жизни в ").text)
```

##### Для QuantizationType.Q5_0 и ContainerType.GGJT
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_0-ggjt.bin")
print(model.generate("Смысл жизни в ").text)
```

##### Для QuantizationType.Q5_0 и ContainerType.GGML
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_0.bin")
print(model.generate("Смысл жизни в ").text)
```

##### Для QuantizationType.Q5_1 и ContainerType.GGJT
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_1-ggjt.bin")
print(model.generate("Смысл жизни в ").text)
```

##### Для QuantizationType.Q5_1 и ContainerType.GGML
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q5_1.bin")
print(model.generate("Смысл жизни в ").text)
```

##### Для QuantizationType.Q8_0 и ContainerType.GGJT
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q8_0-ggjt.bin")
print(model.generate("Смысл жизни в ").text)
```

##### Для QuantizationType.Q8_0 и ContainerType.GGML
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-q8_0.bin")
print(model.generate("Смысл жизни в ").text)
```

##### Версия f16
```python
from llm_rs import AutoModel
model = AutoModel.from_pretrained("iashchak/ruGPT-3.5-13B-ggml", model_file="ruGPT-3.5-13B-f16.bin")
print(model.generate("Смысл жизни в ").text)
```

#### Совместимость

Хотя эта модель предназначена для совместимости с любым GGML-совместимым интерфейсом, она не была тщательно протестирована в таких средах. Используйте на свой страх и риск.