Instructions to use meta-llama/Llama-3.2-3B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use meta-llama/Llama-3.2-3B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="meta-llama/Llama-3.2-3B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use meta-llama/Llama-3.2-3B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "meta-llama/Llama-3.2-3B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-3.2-3B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/meta-llama/Llama-3.2-3B-Instruct

SGLang

How to use meta-llama/Llama-3.2-3B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "meta-llama/Llama-3.2-3B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-3.2-3B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "meta-llama/Llama-3.2-3B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-3.2-3B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use meta-llama/Llama-3.2-3B-Instruct with Docker Model Runner:
```
docker model run hf.co/meta-llama/Llama-3.2-3B-Instruct
```

Wrong scale factor?

by fmello93 - opened Sep 20, 2024

Discussion

fmello93

Sep 20, 2024

AFAIK, scaling factor should be 32, but the config says it’s 8.

fmello93

Sep 23, 2024

@pcuenq

daking

Sep 23, 2024

Are you saying it is supposed to be different for the 3.2 models? Because all of the 3.1 models on HF have a factor of 8 as well.

fmello93

Sep 23, 2024

Yes, it’s supposed to be 32 for llama 3.2. I confirmed internally. I guess that there’s someone looking into it

daking

Sep 24, 2024

Possibly related, but I'm getting gibberish from the model when asking for long context output (seems worse with high temperature). e.g. prompt of "write me a very very very long story about frogs. at least 10000 words." It seems to devolve into gibberish and then at some point recovers with "I apologize for the error. It seems that my previous response was corrupted and contained a large amount of nonsensical text. I'll start again with a new story."

fmello93

Sep 24, 2024

It was updated. Would you mind trying again? @daking

daking

Sep 24, 2024

•

edited Sep 24, 2024

@fmello93 I still get gibberish. Only at very high temperatures (>0.95). Its very strange. It generates gibberish, and then at some point "realizes" and says something along the lines of "I apologize for the error. It seems that my previous response was corrupted and contained a large amount of nonsensical text. I'll start again with a new story." and then carries on with the story.

Also, should the scaling factor update be made to the base models as well?

daking

Sep 24, 2024

Also, to be clear, I'm using the model enablement code in the private wheel of llama-models, so I updated the scaling factor (which was hardcoded as 8) there.

daking

Sep 24, 2024

The update to 32 definitely help, still occasionally results in gibberish with high temperatures, but seems less. Also @fmello93 should the base model configs also be updated to have a scaling factor of 32?

Sanyam changed discussion status to closed Sep 24, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment