Issue with safetensors and memory
#1
by
DevNCT
- opened
AttributeError: 'NoneType' object has no attribute 'endswith'
I probably run out of memory or something is wrong with loading
the Q3_K_S model on my laptop with 16GB RAM and 6GB VRAM
model = AutoModelForCausalLM.from_pretrained(
repo_id,
gguf_file=filename,
trust_remote_code=True,
device_map="auto",
use_cache=True,
max_memory={"cpu": f"{ram:.2f}GiB", 0: f"{vram:.2f}GiB"},
torch_dtype=torch.float
)
File c:\Users\user\anaconda3\envs\llm\lib\site-packages\transformers\modeling_utils.py:4359, in PreTrainedModel._load_pretrained_model(cls, model, state_dict, loaded_keys, resolved_archive_file, pretrained_model_name_or_path, ignore_mismatched_sizes, sharded_metadata, _fast_init, low_cpu_mem_usage, device_map, offload_folder, offload_state_dict, dtype, hf_quantizer, keep_in_fp32_modules, gguf_path, weights_only)
4355 if device_map is not None and "disk" in device_map.values():
4356 archive_file = (
4357 resolved_archive_file[0] if isinstance(resolved_archive_file, (list, tuple)) else resolved_archive_file
4358 )
-> 4359 is_safetensors = archive_file.endswith(".safetensors")
4360 if offload_folder is None and not is_safetensors:
4361 raise ValueError(
4362 "The current `device_map` had weights offloaded to the disk. Please provide an `offload_folder`"
4363 " for them. Alternatively, make sure you have `safetensors` installed if the model you are using"
4364 " offers the weights in this format."
4365 )
AttributeError: 'NoneType' object has no attribute 'endswith'
Any idea?