recourses needed to run this efficiently?
#7
by
Daemontatox
- opened
I can run the base Llama3.2 11b on single Nvidia L4 (24 vram) , whats the recommended amount to run this model efficiently?
I think you can run this model using the same resource as the base model. Feel free to contact me if you encounter any issues!
for anyone trying this out, running on L4 is very slow (<10 tok/ sec). The speed on the HF space is much faster.
@nudelbrot ofcourse the one on huggingface is faster its a rotating A100.
@Xkev thank you , i was making , because when i was trying to host it on huggingface, it kept saying L4 is too low of a memory and i thought the COT requires more VRAM.
Daemontatox
changed discussion status to
closed