Instructions to use nassersala/bloom-small-166 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nassersala/bloom-small-166 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nassersala/bloom-small-166")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nassersala/bloom-small-166")
model = AutoModelForCausalLM.from_pretrained("nassersala/bloom-small-166")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use nassersala/bloom-small-166 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nassersala/bloom-small-166"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nassersala/bloom-small-166",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/nassersala/bloom-small-166

SGLang

How to use nassersala/bloom-small-166 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nassersala/bloom-small-166" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nassersala/bloom-small-166",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nassersala/bloom-small-166" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nassersala/bloom-small-166",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use nassersala/bloom-small-166 with Docker Model Runner:
```
docker model run hf.co/nassersala/bloom-small-166
```

Small BLOOM Model for Functional Testing

Description

I've reduced the size bloom to roughly 0.5GB

This repository hosts a significantly smaller version of the BLOOM model, designed primarily for functional testing purposes. It is an ideal choice for scenarios where computational efficiency and quick iterations are necessary, such as in development and testing environments.

Model Details

The original BLOOM model has been scaled down with the following changes:

Number of Layers (n_layer): Reduced to 12 from the original 70 layers.
Hidden Size (hidden_size): Decreased to 512 from the original 14336.
Number of Attention Heads (n_head): Lowered to 8 from the original 112 heads.

Intended Use

This model is suitable for functional testing and development purposes. It is not recommended for tasks that require high accuracy or complex language understanding and generation capabilities.

Disclaimer

Please note that due to the significant reductions in size and complexity, this model does not retain the full capabilities of the original BLOOM model. Expect limitations in accuracy and depth of language understanding.

crafted by Nasser Ali Alzahrani (@nassersala)

Downloads last month: 13

Safetensors

Model size

0.2B params

Tensor type

F32