aws-neuron

/

optimum-neuron-cache

like 13

Follow

AWS Inferentia and Trainium 62

The inference cache is only available for causal lm models for now. cc @Jingya

We do not have seamless-m4t-v2 support yet, not even in optimum main, so we will need to add support for its export and inference first. Besides regarding the size of this model, we might need tp support for it as well...