Deployment to Inference Endpoints
Hello, has anyone been able to deploy it direct to HF dedicated Inference Endpoints? I get the following error:
{"timestamp":"2024-07-24T08:56:30.300816Z","level":"ERROR","fields":{"message":"Shard complete standard error output:\n\nThe tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. \nThe tokenizer class you load from this checkpoint is 'PreTrainedTokenizerFast'. \nThe class this function is called from is 'LlamaTokenizer'.\nYou are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\nSpecial tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\nTraceback (most recent call last):\n\n File "/opt/conda/bin/text-generation-server", line 8, in \n sys.exit(app())\n\n File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 90, in serve\n server.serve(\n\n File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 253, in serve\n asyncio.run(\n\n File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run\n return loop.run_until_complete(main)\n\n File ...
Is there anything one can to do make it on an Inference Endpoint at this point? I tried GPU · Nvidia L4 · 4x GPUs · 96 GB, with TGI, on all settings left on default.
Best,
Stan
Hi,
I got the same issue,
@stmackcat
, did you resolve it?
this issue needs to get attention...
Yep, this is very urgent. Can't deploy it on Sagemaker. Any workaround?
Any update on this issue?
Hi all! Thanks for reporting and very sorry for the wait. We are working on a fix for easy deployment of meta-llama/Meta-Llama-3.1-8B-Instruct in Inference Endpoints -- in the meantime, please ensure the container URI points to ghcr.io/huggingface/text-generation-inference:2.2.0, the latest version of TGI. For example, if your Endpoint is already created and failed status, you can change the container URI in the UI in the settings of the Endpoint under Container Configuration and selecting 'Custom'.
If you're deploying a new endpoint, this can be accomplished under the 'Advanced Configuration' tab here: https://ui.endpoints.huggingface.co/new. You'll select 'Custom' here as well, then update the container URI to ghcr.io/huggingface/text-generation-inference:2.2.0.
You'll also want to pass this env variable to the endpoint: MODEL_ID=/repository, which will allow you to use test the model using the widget once successfully deployed. I'm attaching a screenshot just in case.
We're actively working on a fix for easier deployment of this model, but in the meantime please let me know if you have additional questions!