Gonsoo/AWS-NeuronCC-2-14-llama-3-Korean-Bllossom-8B

Model Card for Model ID

This model is an AWS Neuron compiled version, neuron-cc 2.14, of the Korean fine-tuned model MLP-KTLim/llama-3-Korean-Bllossom-8B, available at https://huggingface.co/MLP-KTLim/llama-3-Korean-Bllossom-8B. It is intended for deployment on Amazon EC2 Inferentia2 and Amazon SageMaker. For detailed information about the model and its license, please refer to the original MLP-KTLim/llama-3-Korean-Bllossom-8B model page

Model Details

This model is compiled with neuronx-cc version, 2.14 It can be deployed with [v1.0-hf-tgi-0.0.24-pt-2.1.2-inf-neuronx-py310](https://github.com/aws/deep-learning-containers/releases?q=tgi+AND+neuronx&expanded=true on SageMaker Endpoint because this inference docker image is only used on SageMaker

This model can be deployed to Amazon SageMaker Endtpoint with this guide, S3 에 저장된 모델을 SageMaker INF2 에 배포하기

In order to do neuron-compilation and depoly in detail , you can refer to Amazon ECR 의 도커 이미지 기반하에 Amazon EC2 Inferentia2 서빙하기

Hardware

At a minimum hardware, you can use Amazon EC2 inf2.xlarge and more powerful family such as inf2.8xlarge, inf2.24xlarge and inf2.48xlarge. The detailed information is Amazon EC2 Inf2 Instances

Model Card Contact

Gonsoo Moon, gonsoomoon@gmail.com