metadata
library_name: pruna-engine
thumbnail: >-
https://assets-global.website-files.com/646b351987a8d8ce158d1940/64ec9e96b4334c0e1ac41504_Logo%20with%20white%20text.svg
metrics:
- memory_disk
- memory_inference
- inference_latency
- inference_throughput
- inference_CO2_emissions
- inference_energy_consumption
Simply make AI models cheaper, smaller, faster, and greener!
- Give a thumbs up if you like this model!
- Contact us and tell us which model to compress next here.
- Request access to easily compress your own AI models here.
- Read the documentations to know more here
- Join Pruna AI community on Discord here to share feedback/suggestions or get help.
Frequently Asked Questions
- How does the compression work? The model is compressed by using bitsandbytes.
- How does the model quality change? The quality of the model output will slightly degrade.
- What is the model format? We the standard safetensors format.
- How to compress my own models? You can request premium access to more compression methods and tech support for your specific use-cases here.
Usage
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
import torch
from PIL import Image
import requests
processor = LlavaNextProcessor.from_pretrained("PrunaAI/llava-v1.6-vicuna-7b-bnb-8bit")
model = LlavaNextForConditionalGeneration.from_pretrained("PrunaAI/llava-v1.6-vicuna-7b-bnb-8bit")
# prepare image and text prompt, using the appropriate prompt template
url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw)
prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\nWhat is shown in this image? ASSISTANT:"
inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
# autoregressively complete prompt
output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))
Credits & License
The license of the smashed model follows the license of the original model. Please check the license of the original model liuhaotian/llava-v1.6-vicuna-7b before using this model which provided the base model. The license of the pruna-engine is here on Pypi.