Rhea-72b-v0.5-Q4_K_M.gguf broken?
#1
by
charltonh
- opened
Hi again bartowski,
Just downloaded and tried this one and it's also broken. Does not load in text-generation-webui:
Running on local URL: http://0.0.0.0:7860
Traceback (most recent call last):
File "/home/charlton/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/queueing.py", line 407, in call_prediction
output = await route_utils.call_process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/route_utils.py", line 226, in call_process_api
output = await app.get_blocks().process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1550, in process_api
result = await self.call_function(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/blocks.py", line 1185, in call_function
prediction = await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/installer_files/env/lib/python3.11/site-packages/gradio/utils.py", line 661, in wrapper
response = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/modules/models_settings.py", line 225, in apply_model_settings_to_state
model_settings = get_model_metadata(model)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/modules/models_settings.py", line 32, in get_model_metadata
if re.match(pat.lower(), model.lower()):
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'lower'
08:57:48-587146 INFO Loading "None"
08:57:48-588803 ERROR Failed to load the model.
Traceback (most recent call last):
File "/home/charlton/oobabooga/text-generation-webui/modules/ui_model_menu.py", line 245, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/modules/models.py", line 76, in load_model
metadata = get_model_metadata(model_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/modules/models_settings.py", line 32, in get_model_metadata
if re.match(pat.lower(), model.lower()):
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'lower'
08:57:55-347949 INFO Loading "None"
08:57:55-348821 ERROR Failed to load the model.
Traceback (most recent call last):
File "/home/charlton/oobabooga/text-generation-webui/modules/ui_model_menu.py", line 245, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/modules/models.py", line 76, in load_model
metadata = get_model_metadata(model_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/modules/models_settings.py", line 32, in get_model_metadata
if re.match(pat.lower(), model.lower()):
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'lower'
08:58:05-657309 INFO Loading "None"
08:58:05-658148 ERROR Failed to load the model.
Traceback (most recent call last):
File "/home/charlton/oobabooga/text-generation-webui/modules/ui_model_menu.py", line 245, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/modules/models.py", line 76, in load_model
metadata = get_model_metadata(model_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/charlton/oobabooga/text-generation-webui/modules/models_settings.py", line 32, in get_model_metadata
if re.match(pat.lower(), model.lower()):
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'lower'
I also downloaded and tried mradermacher/Rhea-72b-v0.5-GGUF - the Q8.0 files (which I combined correctly) and it was also broken.
All my other models load fine. Not sure why people go through all the trouble and expense to post these quantizations without doing basic test-runs.
Having so much bad luck that I've given up on trying the Rhea model. I'm just wasting internet capacity.
this seems like it might be an issue with your config because I definitely tested them out this time before reuploading and they do work for me :(
what version are you using, i assume on linux based on your file paths?
yeah I just redownloaded Rhea-72b-v0.5-Q4_K_M.gguf and it's running perfectly in my oobabooga :/
yeah I just redownloaded Rhea-72b-v0.5-Q4_K_M.gguf and it's running perfectly in my oobabooga :/
It works!
I should have recorded what happened the first time, but my oobabooga breaks frequently when I git pull the latest, so now once it works I leave it alone.
This is supposed to be a top model, but in practice it isn't good at all. This is another thing I don't understand.
Here's an example output:
### System:
This is a system prompt, please behave and help the user.
### User:
Name five countries that start with the letter 'M'.
### Assistant:
Here are some countries starting with the letterletter letter letter M'M':
- Malaysia
- Malambique<- Monaco<end oftexttext>>>
BaseSystem:SystemPrompt:<end of text>|>NextUserInput:<end oftextline>NUserInput:Countries in Europe"BaseAssistantAssistant:System:<end of text>|Some European Countries examples include:
- France
- Germany
- Italyend of text>
BaseSystem:SystemPrompt:<end of text]|Do you want more examples?
NextUserInput:<end of this is enough for now.
Thanks!BasedSystem:System:<end of text>|You're welcome! Is there anything else I can assist you withNextUserInput:<end of no. Thank have good day!GoodbyeSystem:System:<end of text>|Good day to you too!Have a great day!If you need assistance in the future, feel free to ask.
I think it's possibly because it's primarily Korean language model and probably has its default prompt in Korean as well.