Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference
•
41
Hugo Larcher
hlarcher
AI & ML interests
None yet
Recent Activity
posted
an
update
2 days ago
We are introducing multi-backend support in Hugging Face Text Generation Inference!
With new TGI architecture we are now able to plug new modeling backends to get best performances according to selected model and available hardware. This first step will very soon be followed by the integration of new backends (TRT-LLM, llama.cpp, vLLM, Neuron and TPU).
We are polishing the TensorRT-LLM backend which achieves impressive performances on NVIDIA GPUs, stay tuned 🤗 !
Check out the details: https://huggingface.co/blog/tgi-multi-backend
upvoted
an
article
2 days ago
Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference
updated
a dataset
4 days ago
huggingface/documentation-images
Articles
Organizations
Posts
1
Post
912
We are introducing multi-backend support in Hugging Face Text Generation Inference!
With new TGI architecture we are now able to plug new modeling backends to get best performances according to selected model and available hardware. This first step will very soon be followed by the integration of new backends (TRT-LLM, llama.cpp, vLLM, Neuron and TPU).
We are polishing the TensorRT-LLM backend which achieves impressive performances on NVIDIA GPUs, stay tuned 🤗 !
Check out the details: https://huggingface.co/blog/tgi-multi-backend
With new TGI architecture we are now able to plug new modeling backends to get best performances according to selected model and available hardware. This first step will very soon be followed by the integration of new backends (TRT-LLM, llama.cpp, vLLM, Neuron and TPU).
We are polishing the TensorRT-LLM backend which achieves impressive performances on NVIDIA GPUs, stay tuned 🤗 !
Check out the details: https://huggingface.co/blog/tgi-multi-backend