Issue with safetensors and memory

#1
by DevNCT - opened

AttributeError: 'NoneType' object has no attribute 'endswith'

I probably run out of memory or something is wrong with loading
the Q3_K_S model on my laptop with 16GB RAM and 6GB VRAM

model = AutoModelForCausalLM.from_pretrained(
        repo_id,
        gguf_file=filename,
        trust_remote_code=True,
        device_map="auto",
        use_cache=True,
        max_memory={"cpu": f"{ram:.2f}GiB", 0: f"{vram:.2f}GiB"},
        torch_dtype=torch.float
    )


File c:\Users\user\anaconda3\envs\llm\lib\site-packages\transformers\modeling_utils.py:4359, in PreTrainedModel._load_pretrained_model(cls, model, state_dict, loaded_keys, resolved_archive_file, pretrained_model_name_or_path, ignore_mismatched_sizes, sharded_metadata, _fast_init, low_cpu_mem_usage, device_map, offload_folder, offload_state_dict, dtype, hf_quantizer, keep_in_fp32_modules, gguf_path, weights_only)
   4355 if device_map is not None and "disk" in device_map.values():
   4356     archive_file = (
   4357         resolved_archive_file[0] if isinstance(resolved_archive_file, (list, tuple)) else resolved_archive_file
   4358     )
-> 4359     is_safetensors = archive_file.endswith(".safetensors")
   4360     if offload_folder is None and not is_safetensors:
   4361         raise ValueError(
   4362             "The current `device_map` had weights offloaded to the disk. Please provide an `offload_folder`"
   4363             " for them. Alternatively, make sure you have `safetensors` installed if the model you are using"
   4364             " offers the weights in this format."
   4365         )

AttributeError: 'NoneType' object has no attribute 'endswith'

Any idea?

Sign up or log in to comment