Update README.md

README.md

@@ -19,14 +19,13 @@ license: other
 
 # Falcon 40B-Instruct GGML GGML
 
-These files are GGML format model files for [Falcon 40B-Instruct GGML](https://huggingface.co/tiiuae/falcon-40b-instruct).
+These files are **experimental** GGML format model files for [Falcon 40B-Instruct GGML](https://huggingface.co/tiiuae/falcon-40b-instruct).
 
-GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
-* [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
-* [KoboldCpp](https://github.com/LostRuins/koboldcpp)
-* [ParisNeo/GPT4All-UI](https://github.com/ParisNeo/gpt4all-ui)
-* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
-* [ctransformers](https://github.com/marella/ctransformers)
+At the time of writing these GGML files will **not** work in llama.cpp, or any UI or library.
+
+They can currently only work using the basic command line test tool from a fork of the ggml repo.
+
+They are therefore uploaded purely for initial evaluation and experimentation. Support for these GGMLs should improve in the near future.
 
 ## Repositories available
 
@@ -37,29 +36,20 @@ GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/gger
 <!-- compatibility_ggml start -->
 ## Compatibility
 
-### Original llama.cpp quant methods: `q4_0, q4_1, q5_0, q5_1, q8_0`
-
-I have quantized these 'original' quantisation methods using an older version of llama.cpp so that they remain compatible with llama.cpp as of May 19th, commit `2d5db48`.
-
-They should be compatible with all current UIs and libraries that use llama.cpp, such as those listed at the top of this README.
-
-### New k-quant methods: `q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K`
-
-These new quantisation methods are only compatible with llama.cpp as of June 6th, commit `2d43387`.
+To build the CLI tool necessary to use these GGML files, please follow the following steps:
 
-They will NOT be compatible with koboldcpp, text-generation-ui, and other UIs and libraries yet. Support is expected to come over the next few days.
-
-## Explanation of the new k-quant methods
+```
+git clone https://github.com/jploski/ggml falcon-ggml
+cd falcon-ggml
+git checkout falcon40b
+mkdir build && cd build && cmake .. && cmake --build . --config Release
+```
 
-The new methods available are:
-* GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
-* GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
-* GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
-* GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
-* GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
-* GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type.
+Then run:
+```
+bin/falcon -m /workspace/process/wizard-falcon40b/ggml/Falcon-40b-Instruct.ggmlv3.q4_0.bin -t 10 -n 200 -p "write a story about llamas"
+```
 
-Refer to the Provided Files table below to see what files use which methods, and how.
 <!-- compatibility_ggml end -->
 
 ## Provided files
@@ -70,25 +60,7 @@ Refer to the Provided Files table below to see what files use which methods, and
 | Falcon-40b-Instruct.ggmlv3.q5_0.bin | q5_0 | 5 | 28.77 GB | 31.27 GB | Original llama.cpp quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
 | Falcon-40b-Instruct.ggmlv3.q5_1.bin | q5_1 | 5 | 31.38 GB | 33.88 GB | Original llama.cpp quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
 
-
-**Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
-
-## How to run in `llama.cpp`
-
-I use the following command line; adjust for your tastes and needs:
-
-```
-./main -t 10 -ngl 32 -m wizardcoder-15b.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"
-```
-Change `-t 10` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`.
-
-Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
-
-If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
-
-## How to run in `text-generation-webui`
-
-Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md).
+A q8_0 file will be provided shortly. There is currently an issue preventing it from working. Once this is fixed, it will be uploaded.
 
 <!-- footer start -->
 ## Discord