nguyenthanhdo/mMiniLMv2-L12-H384-distilled-from-XLMR-Large
Updated
•
5
Nguyễn Thành Đô
nguyenthanhdo
AI & ML interests
I'm building open source LLMs that work for Vietnamese
Recent Activity
updated
a model
16 days ago
nguyenthanhdo/mMiniLMv2-L12-H384-distilled-from-XLMR-Large
Organizations
nguyenthanhdo's activity
updated
a
model
16 days ago
updated
3
models
3 months ago
updated
a
dataset
4 months ago
updated
7
models
5 months ago
nguyenthanhdo/ViMath-PAL-Qwen2-7B-LORA
Updated
•
1
nguyenthanhdo/ViMath-PAL-CodeQwen1.5-7B-LORA
Updated
nguyenthanhdo/ViMath-PAL-Llama-3-8B-LORA
Updated
nguyenthanhdo/ViMath-PAL-deepseek-math-7B-LORA
Updated
•
2
nguyenthanhdo/ViMath-CodeQwen1.5-7B-LORA
Updated
nguyenthanhdo/ViMath-Qwen2-7B-LORA
Updated
•
5
nguyenthanhdo/ViMath-Llama-3-8B-LORA
Updated
•
2
liked
a
model
5 months ago
liked
2
datasets
6 months ago
Missing characters' descriptions.
2
#2 opened 6 months ago
by
nguyenthanhdo
nguyenthanhdo
Reacted to
mlabonne's
post with ❤️
6 months ago
Post
9239
⚡ AutoQuant
AutoQuant is the evolution of my previous AutoGGUF notebook (https://colab.research.google.com/drive/1P646NEg33BZy4BfLDNpTz0V0lwIU3CHu). It allows you to quantize your models in five different formats:
- GGUF: perfect for inference on CPUs (and LM Studio)
- GPTQ/EXL2: fast inference on GPUs
- AWQ: super fast inference on GPUs with vLLM (https://github.com/vllm-project/vllm)
- HQQ: extreme quantization with decent 2-bit and 3-bit models
Once the model is converted, it automatically uploads it on the Hugging Face Hub. To quantize a 7B model, GGUF only needs a T4 GPU, while the other methods require an A100 GPU.
Here's an example of a model I quantized using HQQ and AutoQuant: mlabonne/AlphaMonarch-7B-2bit-HQQ
I hope you'll enjoy it and quantize lots of models! :)
💻 AutoQuant: https://colab.research.google.com/drive/1b6nqC7UZVt8bx4MksX7s656GXPM-eWw4
AutoQuant is the evolution of my previous AutoGGUF notebook (https://colab.research.google.com/drive/1P646NEg33BZy4BfLDNpTz0V0lwIU3CHu). It allows you to quantize your models in five different formats:
- GGUF: perfect for inference on CPUs (and LM Studio)
- GPTQ/EXL2: fast inference on GPUs
- AWQ: super fast inference on GPUs with vLLM (https://github.com/vllm-project/vllm)
- HQQ: extreme quantization with decent 2-bit and 3-bit models
Once the model is converted, it automatically uploads it on the Hugging Face Hub. To quantize a 7B model, GGUF only needs a T4 GPU, while the other methods require an A100 GPU.
Here's an example of a model I quantized using HQQ and AutoQuant: mlabonne/AlphaMonarch-7B-2bit-HQQ
I hope you'll enjoy it and quantize lots of models! :)
💻 AutoQuant: https://colab.research.google.com/drive/1b6nqC7UZVt8bx4MksX7s656GXPM-eWw4
- 15 replies
updated
a
dataset
7 months ago
liked
a
dataset
7 months ago
liked
a
model
7 months ago
