metadata
license: llama3
tags:
- distributed-inference
- text-generation
This is converted Llama 3 8B model to Distributed Llama format. The model is quantized to Q40.
🚀 How to Run?
- ⏬ Download this repository.
- ⏬ Download Distributed Llama repository.
- 🔨 Build Distributed Llama:
make dllama
- 🚀 Run Distributed Llama:
./dllama inference --prompt "Hello world" --model dllama_model_meta-llama-3-8b_q40.m --tokenizer dllama_tokenizer_llama3.t --buffer-float-type q80 --nthreads 4