smol_llama-81M-tied

A small 81M param (total) decoder model, enabled through tying the input/output embeddings. This is the first version of the model.

• 768 hidden size, 6 layers
• standard multi-head attention (24 heads), context length 1024
• input/output embeddings are tied
• train-from-scratch

Notes

This checkpoint is the 'raw' pre-trained model and has not been tuned to a more specific task. It should be fine-tuned before use in most cases.

• slightly larger 101M param GQA pretrained version: here
• For the chat version of this model, please see here

---

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	24.52
ARC (25-shot)	22.18
HellaSwag (10-shot)	29.33
MMLU (5-shot)	24.06
TruthfulQA (0-shot)	43.97
Winogrande (5-shot)	49.25
GSM8K (5-shot)	0.23
DROP (3-shot)	2.64