Installation Guide

Welcome to the installation guide for the bitsandbytes library! This document provides step-by-step instructions to install bitsandbytes across various platforms and hardware configurations. The library primarily supports CUDA-based GPUs, but the team is actively working on enabling support for additional backends like AMD ROCm, Intel, and Apple Silicon.

Table of Contents

• CUDA
  • Installation via PyPI
  • Compile from Source
• Multi-backend Support (Alpha Release)
  • Supported Backends
  • Pre-requisites
  • Installation
  • Compile from Source
• PyTorch CUDA Versions

CUDA

bitsandbytes is currently only supported on CUDA GPUs for CUDA versions 11.0 - 12.6. However, there’s an ongoing multi-backend effort under development, which is currently in alpha. If you’re interested in providing feedback or testing, check out the multi-backend section below.

Supported CUDA Configurations

The latest version of the distributed bitsandbytes package is built with the following configurations:

For CUDA systems, ensure your hardware meets the following requirements:

pip install bitsandbytes

pip install pre-built wheel from latest main commit

If you would like to use new feature even before they are officially released and help us test them, feel free to install the wheel directly from our CI (the wheel links will remain stable!):

# Note, if you don't want to reinstall BNBs dependencies, append the `--no-deps` flag!
pip install --force-reinstall 'https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_main/bitsandbytes-0.44.2.dev0-py3-none-manylinux_2_24_x86_64.whl'

Compile from source

For Linux and Windows systems, compiling from source allows you to customize the build configurations. See below for detailed platform-specific instructions (see the CMakeLists.txt if you want to check the specifics and explore some additional options):

apt-get install -y build-essential cmake

git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git && cd bitsandbytes/
pip install -r requirements-dev.txt
cmake -DCOMPUTE_BACKEND=cuda -S .
make
pip install -e .   # `-e` for "editable" install, when developing BNB (otherwise leave that out)

PyTorch CUDA versions

Some bitsandbytes features may need a newer CUDA version than the one currently supported by PyTorch binaries from Conda and pip. In this case, you should follow these instructions to load a precompiled bitsandbytes binary.

1. Determine the path of the CUDA version you want to use. Common paths include:

• /usr/local/cuda
• /usr/local/cuda-XX.X where XX.X is the CUDA version number

Then locally install the CUDA version you need with this script from bitsandbytes:

wget https://raw.githubusercontent.com/bitsandbytes-foundation/bitsandbytes/main/install_cuda.sh
# Syntax cuda_install CUDA_VERSION INSTALL_PREFIX EXPORT_TO_BASH
#   CUDA_VERSION in {110, 111, 112, 113, 114, 115, 116, 117, 118, 120, 121, 122, 123, 124, 125, 126}
#   EXPORT_TO_BASH in {0, 1} with 0=False and 1=True

# For example, the following installs CUDA 12.6 to ~/local/cuda-12.6 and exports the path to your .bashrc

bash install_cuda.sh 126 ~/local 1

2. Set the environment variables BNB_CUDA_VERSION and LD_LIBRARY_PATH by manually overriding the CUDA version installed by PyTorch.

export BNB_CUDA_VERSION=<VERSION>
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<PATH>

For example, to use a local install path:

export BNB_CUDA_VERSION=126
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/YOUR_USERNAME/local/cuda-12.6

3. Now when you launch bitsandbytes with these environment variables, the PyTorch CUDA version is overridden by the new CUDA version (in this example, version 11.7) and a different bitsandbytes library is loaded.

Multi-backend Support (Alpha Release)

Link to give us feedback (bugs, install issues, perf results, requests, etc.):

Supported Backends

For each supported backend, follow the respective instructions below:

Pre-requisites

To use bitsandbytes non-CUDA backends, be sure to install:

pip install "transformers>=4.45.1"

# Create a docker container with latest ROCm image, which includes ROCm libraries
docker pull rocm/dev-ubuntu-22.04:6.1.2-complete
docker run -it --device=/dev/kfd --device=/dev/dri --group-add video rocm/dev-ubuntu-22.04:6.1.2-complete
apt-get update && apt-get install -y git && cd home

# Install pytorch compatible with above ROCm version
pip install torch --index-url https://download.pytorch.org/whl/rocm6.1/

Installation

You can install the pre-built wheels for each backend, or compile from source for custom configurations.

Pre-built Wheel Installation (recommended)

# Note, if you don't want to reinstall BNBs dependencies, append the `--no-deps` flag!
pip install --force-reinstall 'https://github.com/bitsandbytes-foundation/bitsandbytes/releases/download/continuous-release_multi-backend-refactor/bitsandbytes-0.44.1.dev0-py3-none-manylinux_2_24_x86_64.whl'

Compile from Source

# Install bitsandbytes from source
# Clone bitsandbytes repo, ROCm backend is currently enabled on multi-backend-refactor branch
git clone -b multi-backend-refactor https://github.com/bitsandbytes-foundation/bitsandbytes.git && cd bitsandbytes/

# Install dependencies
pip install -r requirements-dev.txt

# Compile & install
apt-get install -y build-essential cmake  # install build tools dependencies, unless present
cmake -DCOMPUTE_BACKEND=hip -S .  # Use -DBNB_ROCM_ARCH="gfx90a;gfx942" to target specific gpu arch
make
pip install -e .   # `-e` for "editable" install, when developing BNB (otherwise leave that out)