metadata

license: mit
datasets:
  - databricks/databricks-dolly-15k
language:
  - en
pipeline_tag: text-generation
tags:
  - dolly
  - dolly-v2
  - instruct
  - sharded
inference: false

dolly-v2-12b: sharded checkpoint

This is a sharded checkpoint (with ~4GB shards) of the databricks/dolly-v2-12b model. Refer to the original model for all details.

this enables low-RAM loading, i.e. Colab :)

Basic Usage

install transformers, accelerate, and bitsandbytes.

pip install -U -q transformers bitsandbytes accelerate

Load the model in 8bit, then run inference:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "ethzanalytics/dolly-v2-12b-sharded"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
          model_name, load_in_8bit=True, device_map="auto",
        )