Llama-2 ONNX

This repository contains optimized version of Llama-2 13B.

Downloading the model

You can use huggingface_hub to download this repository. This can be done through both python scripting and the commandline. Refer to the HuggingFace Hub Documentation for the Python examples.

With CLI:

Make sure you have an updated huggingface_hub installed.

pip install -U huggingface_hub

Download the repository.

huggingface-cli download alpindale/Llama-2-13b-ONNX --repo-type model --cache-dir /path/to/custom/cache/directory --local-dir /path/to/download/dir --local-dir-use-symlinks False

The --cache-dir kwarg is only necessary if your default cache directory (~/.cache) does not have enough disk space to accomodate the entire repository.

Using the model

The repository provides example code for running the models.

python llama2_onnx_inference.py --onnx_file FP16/LlamaV2_13B_float16.onnx --embedding_file embeddings.pth --tokenizer_path tokenizer.model --prompt "What is the lightest element?"

Output:

The lightest element is hydrogen. Hydrogen is the lightest element on the periodic table, with an atomic mass of 1.00794 u (unified atomic mass units).

Chat Interface

Alternatively, you can use the Gradio chat interface to run the models.

First, install the required packages:

pip install -r requirements.txt

Set the Python path to the root directory of the repository (necessary for importing the required modules):

export PYTHONPATH=$PYTHONPATH:$(pwd)

Then you can simply run:

python ChatApp/app.py

You can then navigate to http://localhost:7860 on your browser to access the interface.

FAQ

Why is the first inference session slow?

ONNX runtime execution provider might need to generate JIT binaries for the underlying hardware, typically the binary is cache and will be loaded directly in the subsequent runs to reduce the overhead.

Why is FP16 slower than FP32 on my device?

Your device may not support native FP16 math, therefore weights will be cast to FP32 at runtime. Using the FP32 version of the model will avoid the cast overhead.

How do I optimize inference?

It's recommended that inputs/outputs are put on target device to avoid expensive data copies, please refer to the following documentations for details:

I/O Binding | onnxruntime

What generation parameters should I use the model with?

You can perform temperature and top-p sampling with the provided example code. Please refer to Meta's example here.