--max_seq_len 8192 --compress_pos_emb 4 --loader exllama_hf check monkeypatch in ooba ..
for 16384 compress_pos_emb 8
works on 2 a6000 just fine
--max_seq_len 8192 --compress_pos_emb 4 --loader exllama_hf check monkeypatch in ooba ..
for 16384 compress_pos_emb 8
works on 2 a6000 just fine