README.md

---
language:
- en
license: apache-2.0
tags:
- pretrained
datasets:
- Skylion007/openwebtext
- c4
- wikimedia/wikipedia
- tiiuae/falcon-refinedweb
- izumi-lab/open-text-books
- togethercomputer/RedPajama-Data-V2
- databricks/databricks-dolly-15k
- euclaise/reddit-instruct-curated
- CohereForAI/aya_dataset
pipeline_tag: text-generation
widget:
- messages:
  - role: user
    content: Specs of a game about trolls and warriors in a fantasy world.
- messages:
  - role: user
    content: Reducing waste generation is essential to...
- messages:
  - role: user
    content: Water, planet, resource, future
- messages:
  - role: user
    content: Background story of an RPG game about wizards and dragons in a sci-fi
      world. The story takes place in a...
inference:
  parameters:
    max_new_tokens: 250
    do_sample: true
    temperature: 0.65
    top_p: 0.55
    top_k: 35
    repetition_penalty: 1.176
model-index:
- name: Minueza-32M-Base
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: AI2 Reasoning Challenge (25-Shot)
      type: ai2_arc
      config: ARC-Challenge
      split: test
      args:
        num_few_shot: 25
    metrics:
    - type: acc_norm
      value: 21.33
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HellaSwag (10-Shot)
      type: hellaswag
      split: validation
      args:
        num_few_shot: 10
    metrics:
    - type: acc_norm
      value: 26.39
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU (5-Shot)
      type: cais/mmlu
      config: all
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 24.8
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: TruthfulQA (0-shot)
      type: truthful_qa
      config: multiple_choice
      split: validation
      args:
        num_few_shot: 0
    metrics:
    - type: mc2
      value: 47.45
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: Winogrande (5-shot)
      type: winogrande
      config: winogrande_xl
      split: validation
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 53.2
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GSM8k (5-shot)
      type: gsm8k
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 0.38
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/Minueza-32M-Base
      name: Open LLM Leaderboard
---

# Minueza-32M-Base

## Summary

Minueza-32M-Base is a foundation model with 32 million parameters trained from scratch on a large corpus of text in English.

It's available in the following formats: [Safetensors](https://huggingface.co/Felladrin/Minueza-32M-Base), [GGUF](https://huggingface.co/Felladrin/gguf-Minueza-32M-Base), and [ONNX](https://huggingface.co/Felladrin/onnx-Minueza-32M-Base).

And it's being released alongside some fine-tuned versions:
  - [Minueza-32M-UltraChat](https://huggingface.co/Felladrin/Minueza-32M-UltraChat): Trained on a single conversational dataset.
  - [Minueza-32M-Chat](https://huggingface.co/Felladrin/Minueza-32M-Chat): Trained on a mix of conversational datasets.
  - [Minueza-32Mx2-Chat](https://huggingface.co/Felladrin/Minueza-32Mx2-Chat): Sparse Mixture of Experts trained on interleaved conversational datasets.
  - [And more...](https://huggingface.co/models?other=base_model:Felladrin/Minueza-32M-Base)

## Intended Uses

This model was created with the following objectives in mind:

- Run on mobile web browsers via [Transformers.js](https://huggingface.co/docs/transformers.js).
- Run fast on machines without GPU.
- Serve as a base for fine-tunes using ChatML format, hence the two additional special tokens (`<|im_start|>` and `<|im_end|>`) with `<|im_end|>` as default EOS token.
  - ChatML works great for both instruction and chat models, so if all fine-tunes are made following the ChatML pattern, other users might benefit from the easiness of creating merges.

## Datasets

The model was trained on a subset of each of the following non-synthetic datasets:

- [Skylion007/openwebtext](https://huggingface.co/datasets/Skylion007/openwebtext)
- [c4](https://huggingface.co/datasets/c4)
- [wikimedia/wikipedia - 20231101.simple](https://huggingface.co/datasets/wikimedia/wikipedia/viewer/20231101.simple)
- [tiiuae/falcon-refinedweb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)
- [izumi-lab/open-text-books](https://huggingface.co/datasets/izumi-lab/open-text-books)
- [togethercomputer/RedPajama-Data-V2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2)
- [databricks/databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
- [euclaise/reddit-instruct-curated](https://huggingface.co/datasets/euclaise/reddit-instruct-curated)
- [CohereForAI/aya_dataset - original english annotations](https://huggingface.co/datasets/CohereForAI/aya_dataset/viewer/default/train?f[language_code][value]=%27eng%27)

The subsets were interleaved to form the final training corpus of approximately 650 million tokens.

## Model Architecture

This is a transformer model with the Mistral architecture, trained on a context window of 2048 tokens.

| Configuration           | Value |
| :---------------------- | :---- |
| max_position_embeddings | 2048  |
| hidden_size             | 312   |
| intermediate_size       | 1092  |
| num_attention_heads     | 12    |
| num_hidden_layers       | 10    |
| num_key_value_heads     | 4     |
| vocab_size              | 32002 |

The pretraining was made with these hyperparameters and frameworks:

| Hyperparameter              | Value                                         |
| :-------------------------- | :-------------------------------------------- |
| learning_rate               | 5e-05                                         |
| train_batch_size            | 1                                             |
| eval_batch_size             | 1                                             |
| seed                        | 42                                            |
| gradient_accumulation_steps | 8                                             |
| total_train_batch_size      | 8                                             |
| optimizer                   | Adam with betas=(0.9,0.999) and epsilon=1e-08 |
| lr_scheduler_type           | linear                                        |

| Framework    | Version     |
| :----------- | :---------- |
| Transformers | 4.38.0.dev0 |
| Pytorch      | 2.1.2       |
| Datasets     | 2.16.1      |
| Tokenizers   | 0.15.1      |

## Usage

This is just a base model. For your task, you will likely want to perform application-specific fine-tuning as recommended above.

Also note that this model was trained on internet text data, which may contain biases, offensive or inappropriate content, and may produce incorrect or irrelevant responses. No evaluation has been conducted, so use with care.

Having that said, here's how you can run it:

```python
from transformers import pipeline

generate = pipeline("text-generation", "Felladrin/Minueza-32M-Base")

prompt = "The best way to improve your health is"

output = generate(
    prompt,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.72,
    top_p=0.73,
    top_k=50,
    repetition_penalty=1.176,
)

print(output[0]["generated_text"])
```

## [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Felladrin__Minueza-32M-Base)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |28.92|
|AI2 Reasoning Challenge (25-Shot)|21.33|
|HellaSwag (10-Shot)              |26.39|
|MMLU (5-Shot)                    |24.80|
|TruthfulQA (0-shot)              |47.45|
|Winogrande (5-shot)              |53.20|
|GSM8k (5-shot)                   | 0.38|

## License

This model is licensed under the [Apache License 2.0](https://huggingface.co/Felladrin/Minueza-32M-Base/resolve/main/license.txt).