Hugging Face's logo

second-state
/
Llama-3.3-70B-Instruct-GGUF

Text Generation
Transformers
GGUF
PyTorch
llama
facebook
meta
llama-3
Inference Endpoints
conversational
Model card Files Community
apepkuss79 commited on
Commit
078f013
•
1 Parent(s): abac21c

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +14 -13
README.md CHANGED
@@ -80,7 +80,7 @@ tags:
80
  --ctx-size 128000
81
  ```
82
 
83
- <!-- ## Quantized GGUF Models
84
 
85
  | Name | Quant method | Bits | Size | Use case |
86
  | ---- | ---- | ---- | ---- | ----- |
@@ -88,20 +88,21 @@ tags:
88
  | [Llama-3.3-70B-Instruct-Q3_K_L.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q3_K_L.gguf) | Q3_K_L | 3 | 37.1 GB| small, substantial quality loss |
89
  | [Llama-3.3-70B-Instruct-Q3_K_M.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q3_K_M.gguf) | Q3_K_M | 3 | 34.3 GB| very small, high quality loss |
90
  | [Llama-3.3-70B-Instruct-Q3_K_S.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q3_K_S.gguf) | Q3_K_S | 3 | 30.9 GB| very small, high quality loss |
91
- | [Llama-3.3-70B-Instruct-Q4_0.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q4_0.gguf) | Q4_0 | 4 | 40 GB| legacy; small, very high quality loss - prefer using Q3_K_M |
92
  | [Llama-3.3-70B-Instruct-Q4_K_M.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q4_K_M.gguf) | Q4_K_M | 4 | 42.5 GB| medium, balanced quality - recommended |
 
93
  | [Llama-3.3-70B-Instruct-Q5_0.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q5_0.gguf) | Q5_0 | 5 | 48.7 GB| legacy; medium, balanced quality - prefer using Q4_K_M |
94
- | [Llama-3.3-70B-Instruct-Q5_K_M.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q5_K_M.gguf) | Q5_K_M | 5 | 50 GB| large, very low quality loss - recommended |
95
  | [Llama-3.3-70B-Instruct-Q5_K_S.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q5_K_S.gguf) | Q5_K_S | 5 | 48.7 GB| large, low quality loss - recommended |
96
- | [Llama-3.3-70B-Instruct-Q6_K-00001-of-00002.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q6_K-00001-of-00002.gguf) | Q6_K | 6 | 32.1 GB| very large, extremely low quality loss |
97
- | [Llama-3.3-70B-Instruct-Q6_K-00002-of-00002.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q6_K-00002-of-00002.gguf) | Q6_K | 6 | 25.7 GB| very large, extremely low quality loss |
98
- | [Llama-3.3-70B-Instruct-Q8_0-00001-of-00003.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q8_0-00001-of-00003.gguf) | Q8_0 | 8 | 32 GB| very large, extremely low quality loss - not recommended |
99
- | [Llama-3.3-70B-Instruct-Q8_0-00002-of-00003.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q8_0-00002-of-00003.gguf) | Q8_0 | 8 | 32.1 GB| very large, extremely low quality loss - not recommended |
100
- | [Llama-3.3-70B-Instruct-Q8_0-00003-of-00003.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q8_0-00003-of-00003.gguf) | Q8_0 | 8 | 10.9 GB| very large, extremely low quality loss - not recommended |
101
- | [Llama-3.3-70B-Instruct-f16-00001-of-00005.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-f16-00001-of-00005.gguf) | f16 | 16 | 32.1 GB| |
102
- | [Llama-3.3-70B-Instruct-f16-00002-of-00005.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-f16-00002-of-00005.gguf) | f16 | 16 | 32 GB| |
103
- | [Llama-3.3-70B-Instruct-f16-00003-of-00005.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-f16-00003-of-00005.gguf) | f16 | 16 | 32 GB| |
104
- | [Llama-3.3-70B-Instruct-f16-00004-of-00005.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-f16-00004-of-00005.gguf) | f16 | 16 | 31.7 GB| |
105
- | [Llama-3.3-70B-Instruct-f16-00005-of-00005.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-f16-00005-of-00005.gguf) | f16 | 16 | 13.1 GB| | -->
106
 
107
  *Quantized with llama.cpp b4273.*
 
80
  --ctx-size 128000
81
  ```
82
 
83
+ ## Quantized GGUF Models
84
 
85
  | Name | Quant method | Bits | Size | Use case |
86
  | ---- | ---- | ---- | ---- | ----- |
 
88
  | [Llama-3.3-70B-Instruct-Q3_K_L.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q3_K_L.gguf) | Q3_K_L | 3 | 37.1 GB| small, substantial quality loss |
89
  | [Llama-3.3-70B-Instruct-Q3_K_M.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q3_K_M.gguf) | Q3_K_M | 3 | 34.3 GB| very small, high quality loss |
90
  | [Llama-3.3-70B-Instruct-Q3_K_S.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q3_K_S.gguf) | Q3_K_S | 3 | 30.9 GB| very small, high quality loss |
91
+ | [Llama-3.3-70B-Instruct-Q4_0.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q4_0.gguf) | Q4_0 | 4 | 40.0 GB| legacy; small, very high quality loss - prefer using Q3_K_M |
92
  | [Llama-3.3-70B-Instruct-Q4_K_M.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q4_K_M.gguf) | Q4_K_M | 4 | 42.5 GB| medium, balanced quality - recommended |
93
+ | [Llama-3.3-70B-Instruct-Q4_K_S.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q4_K_S.gguf) | Q4_K_S | 4 | 40.3 GB| small, greater quality loss |
94
  | [Llama-3.3-70B-Instruct-Q5_0.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q5_0.gguf) | Q5_0 | 5 | 48.7 GB| legacy; medium, balanced quality - prefer using Q4_K_M |
95
+ | [Llama-3.3-70B-Instruct-Q5_K_M.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q5_K_M.gguf) | Q5_K_M | 5 | 49.9 GB| large, very low quality loss - recommended |
96
  | [Llama-3.3-70B-Instruct-Q5_K_S.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q5_K_S.gguf) | Q5_K_S | 5 | 48.7 GB| large, low quality loss - recommended |
97
+ | [Llama-3.3-70B-Instruct-Q6_K-00001-of-00002.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q6_K-00001-of-00002.gguf) | Q6_K | 6 | 29.9 GB| very large, extremely low quality loss |
98
+ | [Llama-3.3-70B-Instruct-Q6_K-00002-of-00002.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q6_K-00002-of-00002.gguf) | Q6_K | 6 | 28.0 GB| very large, extremely low quality loss |
99
+ | [Llama-3.3-70B-Instruct-Q8_0-00001-of-00003.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q8_0-00001-of-00003.gguf) | Q8_0 | 8 | 29.9 GB| very large, extremely low quality loss - not recommended |
100
+ | [Llama-3.3-70B-Instruct-Q8_0-00002-of-00003.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q8_0-00002-of-00003.gguf) | Q8_0 | 8 | 29.8 GB| very large, extremely low quality loss - not recommended |
101
+ | [Llama-3.3-70B-Instruct-Q8_0-00003-of-00003.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q8_0-00003-of-00003.gguf) | Q8_0 | 8 | 15.3 GB| very large, extremely low quality loss - not recommended |
102
+ | [Llama-3.3-70B-Instruct-f16-00001-of-00005.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-f16-00001-of-00005.gguf) | f16 | 16 | 30.0 GB| |
103
+ | [Llama-3.3-70B-Instruct-f16-00002-of-00005.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-f16-00002-of-00005.gguf) | f16 | 16 | 29.6 GB| |
104
+ | [Llama-3.3-70B-Instruct-f16-00003-of-00005.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-f16-00003-of-00005.gguf) | f16 | 16 | 29.9 GB| |
105
+ | [Llama-3.3-70B-Instruct-f16-00004-of-00005.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-f16-00004-of-00005.gguf) | f16 | 16 | 29.6 GB| |
106
+ | [Llama-3.3-70B-Instruct-f16-00005-of-00005.gguf](https://huggingface.co/second-state/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-f16-00005-of-00005.gguf) | f16 | 16 | 22.2 GB| |
107
 
108
  *Quantized with llama.cpp b4273.*