Dracones/Merged-RP-Stew-V2-34B_exl2_3.5bpw

Merged-RP-Stew-V2-34B - EXL2 3.5bpw

This is a 3.5bpw EXL2 quant of ParasiticRogue/Merged-RP-Stew-V2-34B

Details about the model and the merge info can be found at the above mode page.

EXL2 Version

These quants were made with exllamav2 version 0.0.17. Quants made on this version of EXL2 may not working on older versions of the exllamav2 library.

If you have problems loading these models, please update Text Generation WebUI to the latest version.

Perplexity Scoring

Below are the perplexity scores for the EXL2 models. A lower score is better.

Quant Level	Perplexity Score
8.0	5.2805
7.0	5.2841
6.0	5.2789
5.0	5.2930
4.5	5.3110
4.0	5.3663
3.5	5.4860

EQ Bench

Here are the EQ Bench scores for the EXL2 quants using Alpaca, ChatML and Chat-Vicuna prompt templates. A higher score is better.

Quant Size	ChatML	Alpaca	Chat-Vicuna
8.0	73.82	75.02	73.53
7.0	74.22	74.96	72.71
6.0	74.29	75.11	74.16
5.0	72.74	74.11	73.28
4.5	73.62	73.18	73.32
4.0	73.99	71.85	74.74
3.5	72.07	73.73	73.56

Chat-Vicuna Template

This is the Chat-Vicuna template yaml that was used in EQ bench. It was tested in Text Generation Web UI and seemed to produce accurate results.

user: "USER:"
bot: "ASSISTANT:"
turn_template: "<|user|> <|user-message|><|im_end|>\n<|bot|> <|bot-message|><|im_end|></s>\n"
context: "<|system-message|><|im_end|>\n\n"
system_message: "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions."

Perplexity Script

This was the script used for perplexity testing.

#!/bin/bash

# Activate the conda environment
source ~/miniconda3/etc/profile.d/conda.sh
conda activate exllamav2

# Set the model name
MODEL_NAME="Merged-RP-Stew-V2-34B"
BIT_PRECISIONS=(8.0 7.0 6.0 5.0 4.5 4.0 3.5 3.0 2.65 2.4)

for BIT_PRECISION in "${BIT_PRECISIONS[@]}"
do
    MODEL_DIR="models/${MODEL_NAME}_exl2_${BIT_PRECISION}bpw"

    if [ -d "$MODEL_DIR" ]; then
        output=$(python test_inference.py -m "$MODEL_DIR" -gs 21,24 -ed data/wikitext/wikitext-2-v1.parquet)
        score=$(echo "$output" | grep -oP 'Evaluation perplexity: \K[\d.]+')
        echo "BPW: $BIT_PRECISION, Score: $score"
    fi

done

Quant Details

This is the script used for quantization. These quants were calibrated against Bluemoon-Light's Chat Vicuna training data.

#!/bin/bash

# Activate the conda environment
source ~/miniconda3/etc/profile.d/conda.sh
conda activate exllamav2

# Set the model name and bit size
MODEL_NAME="Merged-RP-Stew-V2-34B"
BIT_PRECISION=3.5

# Define variables
MODEL_DIR="models/$MODEL_NAME"
OUTPUT_DIR="exl2_$MODEL_NAME"
MEASUREMENT_FILE="measurements/$MODEL_NAME.json"
CONVERTED_FOLDER="models/${MODEL_NAME}_exl2_${BIT_PRECISION}bpw"
CALIBRATION_DATASET="data/Bluemoon-Light/chat-vicuna.parquet"

# Create directories
rm -r "$OUTPUT_DIR"
mkdir "$OUTPUT_DIR"
mkdir "$CONVERTED_FOLDER"
  
# Run conversion commands  
python convert.py -i $MODEL_DIR -o $OUTPUT_DIR -nr -om $MEASUREMENT_FILE -c $CALIBRATION_DATASET
python convert.py -i $MODEL_DIR -o $OUTPUT_DIR -nr -m $MEASUREMENT_FILE -b $BIT_PRECISION -c $CALIBRATION_DATASET -cf $CONVERTED_FOLDER