How can I use Sagemaker's inference recommender for this model for question-answering task?
#15
by
monasbhar09
- opened
I have been trying to get the inference recommender work for this model but I do not see a very straight-forward solution. I tried to follow this but this does not work for LLMs I suppose:
https://github.com/aws/amazon-sagemaker-examples/tree/main/sagemaker-inference-recommender/huggingface-inference-recommender
For eg, initializing pipeline for question-answering with OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5 model kills the kernel every time.
And when I get around by adding the model manually, the inference job fails with keyerror: 'gpt_neox' error.
Did anyone try inference recommender for this model and succeeded?