These are the VRAM and RAM requirements (in MiB) to run some examples of models in 16-bit (default) precision:
| model | VRAM (GPU) | RAM |
|---|---|---|
| arxiv_ai_gpt2 | 1512.37 | 5824.2 |
| blenderbot-1B-distill | 2441.75 | 4425.91 |
| opt-1.3b | 2509.61 | 4427.79 |
| gpt-neo-1.3b | 2605.27 | 5851.58 |
| opt-2.7b | 5058.05 | 4863.95 |
| gpt4chan_model_float16 | 11653.7 | 4437.71 |
| gpt-j-6B | 11653.7 | 5633.79 |
| galactica-6.7b | 12697.9 | 4429.89 |
| opt-6.7b | 12700 | 4368.66 |
| bloomz-7b1-p3 | 13483.1 | 4470.34 |
GPU mode with 8-bit precision
Allows you to load models that would not normally fit into your GPU. Enabled by default for 13b and 20b models in this web UI.
| model | VRAM (GPU) | RAM |
|---|---|---|
| opt-13b | 12528.1 | 1152.39 |
| gpt-neox-20b | 20384 | 2291.7 |
CPU mode (32-bit precision)
A lot slower, but does not require a GPU.
On my i5-12400F, 6B models take around 10-20 seconds to respond in chat mode, and around 5 minutes to generate a 200 tokens completion.
| model | RAM |
|---|---|
| arxiv_ai_gpt2 | 4430.82 |
| gpt-neo-1.3b | 6089.31 |
| opt-1.3b | 8411.12 |
| blenderbot-1B-distill | 8508.16 |
| opt-2.7b | 14969.3 |
| bloomz-7b1-p3 | 21371.2 |
| gpt-j-6B | 24200.3 |
| gpt4chan_model | 24246.3 |
| galactica-6.7b | 26561.4 |
| opt-6.7b | 29596.6 |