Instructions to use openchat/openchat_3.5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use openchat/openchat_3.5 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="openchat/openchat_3.5") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("openchat/openchat_3.5") model = AutoModelForCausalLM.from_pretrained("openchat/openchat_3.5") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use openchat/openchat_3.5 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "openchat/openchat_3.5" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "openchat/openchat_3.5", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/openchat/openchat_3.5
- SGLang
How to use openchat/openchat_3.5 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "openchat/openchat_3.5" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "openchat/openchat_3.5", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "openchat/openchat_3.5" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "openchat/openchat_3.5", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use openchat/openchat_3.5 with Docker Model Runner:
docker model run hf.co/openchat/openchat_3.5
Great. But you need to filter out url sources and other gratuitious info.
It not only goes on an on sometimes, but often repeats in endless loops.
For example, after finishing a response it will often start providing a source, which is always wrong by the way, so any url generation directives should be removed from the training data. Anyways, then it might add a warning about respecting anything remotely related to the top, such as minorities, women, celebrities..., then adds a few emojis, then ask if there is anything else, then repeat endlessly.
You need to filter out far more gratuitously additions like moralizing, url references, asking if there's anything else you want, adding emojis... otherwise the end token is never given priority. The alignment tax on this one is making it unusable.
In short, the most important things are respecting the users wishes, brevity and courtesy, and only in rare occassions (such as when asking how to make meth or steal a car) saying no. If something isn't remotely illegal or amoral (e.g. celebrity gossip) then stay out of it. No adult in real life lectures another adult like this. This isn't human alignment. This is brainless and unnatural AI moralizing. I don't care at all about celebrities. They're just test questions of mine because they test for things like this. And this LLM failed miserably. Don't use AI alignment to needless lecture users.
It was better a week ago. I don't know what they changed, but it never used to go on rambling tirades before.
Thanks for clarifying @JJJJJPSYCHIC , I just tried using it recently. I hope they're able to fix the issue. My guess is they just added one too many alignment add-ons until they buried the end token (e.g. please respect celebrities, here's a link, emojis, anything else I can help you with...).