README.md

---
language:
- en
- ko
license: llama3
library_name: transformers
datasets:
- legacy-datasets/wikipedia
pipeline_tag: text-generation
---
## Model Details

This model was continually pretrained from the [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B), using English and Korean datasets.
The goal is to enhance its proficiency in Korean while maintaining its English language capabilities from the original model.

### Datasets

We sampled 16B tokens from the following datasets for training:

<table>
  <tr>
   <td><strong>Sources</strong>
   </td>
   <td><strong>Tokens (Llama-3-8B)</strong>
   </td>
  </tr>
  <tr>
   <td>AI-Hub
   </td>
   <td>9.2B
   </td>
  </tr>
  <tr>
   <td>Modu Corpus
   </td>
   <td>5.8B
   </td>
  </tr>
  <tr>
   <td>Wikipedia
   </td>
   <td>5.4B
   </td>
  </tr>
</table>

### Hyperparameters

<table>
  <tr>
   <td><strong>Learning rate</strong></td>
   <td><strong>Optimizer</strong></td>
   <td><strong>Betas</strong></td>
   <td><strong>Weight decay</strong></td>
   <td><strong>Warm-up ratio</strong></td>
  </tr>
  <tr>
   <td>3e-5</td>
   <td>AdamW</td>
   <td>(0.9, 0.95)</td>
   <td>0.1</td>
   <td>0.05</td>
  </tr>
</table>

## Intended Use

This model has not been fine-tuned, so you will need to train it on your own dataset before using it.

## Evaluations

We evaluated this model using both English and Korean benchmarks, and compared it with similar models that were continually pretrained from the [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B).

<table>
  <tr>
   <td></td>
   <td colspan="4"><strong>English</strong></td>
   <td colspan="3"><strong>Korean</strong></td>
  </tr>
  <tr>
   <td><strong>Model</strong></td>
   <td><strong>MMLU (5-shot)</strong></td>
   <td><strong>HellaSwag (10-shot)</strong></td>
   <td><strong>GSM8K (8-shot, CoT)</strong></td>
   <td><strong>BBH (3-shot, CoT)</strong></td>
   <td><strong>KMMLU (5-shot)</strong></td>
   <td><strong>HAE-RAE (5-shot)</strong></td>
   <td><strong>KoBEST (5-shot)</strong></td>
  </tr>
  <tr>
   <td>meta-llama/Meta-Llama-3-8B</td>
   <td><strong>65.1</strong></td>
   <td><strong>82.1</strong></td>
   <td><strong>52.0</strong></td>
   <td><strong>61.9</strong></td>
   <td>40.2</td>
   <td>61.1</td> 
   <td>69.2</td>  
  </tr>
  <tr>
   <td>saltlux/Ko-Llama3-Luxia-8B</td>
   <td>57.1</td>
   <td>77.1</td>
   <td>32.3</td>
   <td>51.8</td>
   <td>39.4</td>
   <td>69.2</td> 
   <td>71.9</td>  
  </tr>
  <tr>
   <td>beomi/Llama-3-Open-Ko-8B</td>
   <td>56.2</td>
   <td>77.4</td>
   <td>31.5</td>
   <td>46.8</td>
   <td>40.3</td>
   <td>68.1</td> 
   <td><u>72.1</u></td>  
  </tr>
  <tr>
   <td>beomi/Llama-3-KoEn-8B</td>
   <td>52.5</td>
   <td>77.7</td>
   <td>21.2</td>
   <td>43.2</td>
   <td><u>40.8</u></td>
   <td><u>71.3</u></td> 
   <td><strong>73.8</strong></td>  
  </tr>
  <tr>
   <td><strong>tesser-ai/Tesser-Llama-3-Ko-8B</strong></td>
   <td><u>60.5</u></td>
   <td><u>79.8</u></td>
   <td><u>40.3</u></td>
   <td><u>56.3</u></td>
   <td><strong>42.5</strong></td>
   <td><strong>72.1</strong></td> 
   <td><strong>73.8</strong></td>  
  </tr>
</table>

## Limitations

We trained this model using a context length of 4k due to resource limitations and to maximize training speed.
However, the original model was trained with a context length of 8k, so an 8k context length could work well in downstream tasks.

## License

This model follows the original [Llama-3 license](https://llama.meta.com/llama3/license/).