Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference
•
42
Hugo Larcher
hlarcher
AI & ML interests
None yet
Recent Activity
posted
an
update
2 days ago
We are introducing multi-backend support in Hugging Face Text Generation Inference!
With new TGI architecture we are now able to plug new modeling backends to get best performances according to selected model and available hardware. This first step will very soon be followed by the integration of new backends (TRT-LLM, llama.cpp, vLLM, Neuron and TPU).
We are polishing the TensorRT-LLM backend which achieves impressive performances on NVIDIA GPUs, stay tuned 🤗 !
Check out the details: https://huggingface.co/blog/tgi-multi-backend
upvoted
an
article
2 days ago
Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference
updated
a dataset
4 days ago
huggingface/documentation-images
Articles
Organizations
hlarcher's activity
posted
an
update
2 days ago
Post
964
We are introducing multi-backend support in Hugging Face Text Generation Inference!
With new TGI architecture we are now able to plug new modeling backends to get best performances according to selected model and available hardware. This first step will very soon be followed by the integration of new backends (TRT-LLM, llama.cpp, vLLM, Neuron and TPU).
We are polishing the TensorRT-LLM backend which achieves impressive performances on NVIDIA GPUs, stay tuned 🤗 !
Check out the details: https://huggingface.co/blog/tgi-multi-backend
With new TGI architecture we are now able to plug new modeling backends to get best performances according to selected model and available hardware. This first step will very soon be followed by the integration of new backends (TRT-LLM, llama.cpp, vLLM, Neuron and TPU).
We are polishing the TensorRT-LLM backend which achieves impressive performances on NVIDIA GPUs, stay tuned 🤗 !
Check out the details: https://huggingface.co/blog/tgi-multi-backend
upvoted
an
article
2 days ago
Article
Introducing multi-backends (TRT-LLM, vLLM) support for Text Generation Inference
•
42
updated
a
dataset
4 days ago
updated
a
dataset
10 days ago
updated
2
datasets
3 months ago
updated
a
Space
5 months ago
liked
3
Spaces
8 months ago
liked
2
Spaces
10 months ago
upvoted
a
paper
11 months ago
reacted to
loubnabnl's
post with 🤯
11 months ago
Post
⭐ Today we’re releasing The Stack v2 & StarCoder2: a series of 3B, 7B & 15B code generation models trained on 3.3 to 4.5 trillion tokens of code:
- StarCoder2-15B matches or outperforms CodeLlama 34B, and approaches DeepSeek-33B on multiple benchmarks.
- StarCoder2-3B outperforms StarCoderBase-15B and similar sized models.
- The Stack v2 a 4x larger dataset than the Stack v1, resulting in 900B unique code tokens 🚀
As always, we released everything from models and datasets to curation code. Enjoy!
🔗 StarCoder2 collection: bigcode/starcoder2-65de6da6e87db3383572be1a
🔗 Paper: https://drive.google.com/file/d/17iGn3c-sYNiLyRSY-A85QOzgzGnGiVI3/view
🔗 BlogPost: https://huggingface.co/blog/starcoder2
🔗 Code Leaderboard: bigcode/bigcode-models-leaderboard
- StarCoder2-15B matches or outperforms CodeLlama 34B, and approaches DeepSeek-33B on multiple benchmarks.
- StarCoder2-3B outperforms StarCoderBase-15B and similar sized models.
- The Stack v2 a 4x larger dataset than the Stack v1, resulting in 900B unique code tokens 🚀
As always, we released everything from models and datasets to curation code. Enjoy!
🔗 StarCoder2 collection: bigcode/starcoder2-65de6da6e87db3383572be1a
🔗 Paper: https://drive.google.com/file/d/17iGn3c-sYNiLyRSY-A85QOzgzGnGiVI3/view
🔗 BlogPost: https://huggingface.co/blog/starcoder2
🔗 Code Leaderboard: bigcode/bigcode-models-leaderboard
liked
a
model
11 months ago
upvoted
a
paper
over 1 year ago