mradermacher
/

c4ai-command-r-08-2024-i1-GGUF

Transformers

GGUF

Inference Endpoints

Model card Files Files and versions Community

mradermacher commited on 22 days ago

Commit

0f51c16

•

1 Parent(s): 120014d

auto-patch README.md

Browse files

Files changed (1) hide show

README.md +339 -0

README.md CHANGED Viewed

@@ -1,6 +1,345 @@
 <!-- ### quantize_version: 2 -->
 <!-- ### output_tensor_quantised: 1 -->
 <!-- ### convert_type: hf -->
 <!-- ### vocab_type:  -->
 <!-- ### tags: nicoboss -->
 weighted/imatrix quants of https://huggingface.co/CohereForAI/c4ai-command-r-08-2024

+---
+base_model: CohereForAI/c4ai-command-r-08-2024
+extra_gated_fields:
+  Affiliation: text
+  Country:
+    options:
+    - Aruba
+    - Afghanistan
+    - Angola
+    - Anguilla
+    - Åland Islands
+    - Albania
+    - Andorra
+    - United Arab Emirates
+    - Argentina
+    - Armenia
+    - American Samoa
+    - Antarctica
+    - French Southern Territories
+    - Antigua and Barbuda
+    - Australia
+    - Austria
+    - Azerbaijan
+    - Burundi
+    - Belgium
+    - Benin
+    - Bonaire Sint Eustatius and Saba
+    - Burkina Faso
+    - Bangladesh
+    - Bulgaria
+    - Bahrain
+    - Bahamas
+    - Bosnia and Herzegovina
+    - Saint Barthélemy
+    - Belarus
+    - Belize
+    - Bermuda
+    - Plurinational State of Bolivia
+    - Brazil
+    - Barbados
+    - Brunei-Darussalam
+    - Bhutan
+    - Bouvet-Island
+    - Botswana
+    - Central African Republic
+    - Canada
+    - Cocos (Keeling) Islands
+    - Switzerland
+    - Chile
+    - China
+    - Côte-dIvoire
+    - Cameroon
+    - Democratic Republic of the Congo
+    - Cook Islands
+    - Colombia
+    - Comoros
+    - Cabo Verde
+    - Costa Rica
+    - Cuba
+    - Curaçao
+    - Christmas Island
+    - Cayman Islands
+    - Cyprus
+    - Czechia
+    - Germany
+    - Djibouti
+    - Dominica
+    - Denmark
+    - Dominican Republic
+    - Algeria
+    - Ecuador
+    - Egypt
+    - Eritrea
+    - Western Sahara
+    - Spain
+    - Estonia
+    - Ethiopia
+    - Finland
+    - Fiji
+    - Falkland Islands (Malvinas)
+    - France
+    - Faroe Islands
+    - Federated States of Micronesia
+    - Gabon
+    - United Kingdom
+    - Georgia
+    - Guernsey
+    - Ghana
+    - Gibraltar
+    - Guinea
+    - Guadeloupe
+    - Gambia
+    - Guinea Bissau
+    - Equatorial Guinea
+    - Greece
+    - Grenada
+    - Greenland
+    - Guatemala
+    - French Guiana
+    - Guam
+    - Guyana
+    - Hong Kong
+    - Heard Island and McDonald Islands
+    - Honduras
+    - Croatia
+    - Haiti
+    - Hungary
+    - Indonesia
+    - Isle of Man
+    - India
+    - British Indian Ocean Territory
+    - Ireland
+    - Islamic Republic of Iran
+    - Iraq
+    - Iceland
+    - Israel
+    - Italy
+    - Jamaica
+    - Jersey
+    - Jordan
+    - Japan
+    - Kazakhstan
+    - Kenya
+    - Kyrgyzstan
+    - Cambodia
+    - Kiribati
+    - Saint-Kitts-and-Nevis
+    - South Korea
+    - Kuwait
+    - Lao-Peoples-Democratic-Republic
+    - Lebanon
+    - Liberia
+    - Libya
+    - Saint-Lucia
+    - Liechtenstein
+    - Sri Lanka
+    - Lesotho
+    - Lithuania
+    - Luxembourg
+    - Latvia
+    - Macao
+    - Saint Martin (French-part)
+    - Morocco
+    - Monaco
+    - Republic of Moldova
+    - Madagascar
+    - Maldives
+    - Mexico
+    - Marshall Islands
+    - North Macedonia
+    - Mali
+    - Malta
+    - Myanmar
+    - Montenegro
+    - Mongolia
+    - Northern Mariana Islands
+    - Mozambique
+    - Mauritania
+    - Montserrat
+    - Martinique
+    - Mauritius
+    - Malawi
+    - Malaysia
+    - Mayotte
+    - Namibia
+    - New Caledonia
+    - Niger
+    - Norfolk Island
+    - Nigeria
+    - Nicaragua
+    - Niue
+    - Netherlands
+    - Norway
+    - Nepal
+    - Nauru
+    - New Zealand
+    - Oman
+    - Pakistan
+    - Panama
+    - Pitcairn
+    - Peru
+    - Philippines
+    - Palau
+    - Papua New Guinea
+    - Poland
+    - Puerto Rico
+    - North Korea
+    - Portugal
+    - Paraguay
+    - State of Palestine
+    - French Polynesia
+    - Qatar
+    - Réunion
+    - Romania
+    - Russia
+    - Rwanda
+    - Saudi Arabia
+    - Sudan
+    - Senegal
+    - Singapore
+    - South Georgia and the South Sandwich Islands
+    - Saint Helena Ascension and Tristan da Cunha
+    - Svalbard and Jan Mayen
+    - Solomon Islands
+    - Sierra Leone
+    - El Salvador
+    - San Marino
+    - Somalia
+    - Saint Pierre and Miquelon
+    - Serbia
+    - South Sudan
+    - Sao Tome and Principe
+    - Suriname
+    - Slovakia
+    - Slovenia
+    - Sweden
+    - Eswatini
+    - Sint Maarten (Dutch-part)
+    - Seychelles
+    - Syrian Arab Republic
+    - Turks and Caicos Islands
+    - Chad
+    - Togo
+    - Thailand
+    - Tajikistan
+    - Tokelau
+    - Turkmenistan
+    - Timor Leste
+    - Tonga
+    - Trinidad and Tobago
+    - Tunisia
+    - Turkey
+    - Tuvalu
+    - Taiwan
+    - United Republic of Tanzania
+    - Uganda
+    - Ukraine
+    - United States Minor Outlying Islands
+    - Uruguay
+    - United-States
+    - Uzbekistan
+    - Holy See (Vatican City State)
+    - Saint Vincent and the Grenadines
+    - Bolivarian Republic of Venezuela
+    - Virgin Islands British
+    - Virgin Islands U.S.
+    - VietNam
+    - Vanuatu
+    - Wallis and Futuna
+    - Samoa
+    - Yemen
+    - South Africa
+    - Zambia
+    - Zimbabwe
+    type: select
+  I agree to use this model for non-commercial use ONLY: checkbox
+  Name: text
+  Receive email updates on C4AI and Cohere research, events, products and services?:
+    options:
+    - Yes
+    - No
+    type: select
+extra_gated_prompt: By submitting this form, you agree to the [License Agreement](https://cohere.com/c4ai-cc-by-nc-license)  and
+  acknowledge that the information you provide will be collected, used, and shared
+  in accordance with Cohere’s [Privacy Policy]( https://cohere.com/privacy).
+language:
+- en
+- fr
+- de
+- es
+- it
+- pt
+- ja
+- ko
+- zh
+- ar
+library_name: transformers
+license: cc-by-nc-4.0
+quantized_by: mradermacher
+---
+## About
 <!-- ### quantize_version: 2 -->
 <!-- ### output_tensor_quantised: 1 -->
 <!-- ### convert_type: hf -->
 <!-- ### vocab_type:  -->
 <!-- ### tags: nicoboss -->
 weighted/imatrix quants of https://huggingface.co/CohereForAI/c4ai-command-r-08-2024
+<!-- provided-files -->
+static quants are available at https://huggingface.co/mradermacher/c4ai-command-r-08-2024-GGUF
+## Usage
+If you are unsure how to use GGUF files, refer to one of [TheBloke's
+READMEs](https://huggingface.co/TheBloke/KafkaLM-70B-German-V0.1-GGUF) for
+more details, including on how to concatenate multi-part files.
+## Provided Quants
+(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)
+| Link | Type | Size/GB | Notes |
+|:-----|:-----|--------:|:------|
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-IQ1_S.gguf) | i1-IQ1_S | 7.9 | for the desperate |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-IQ1_M.gguf) | i1-IQ1_M | 8.5 | mostly desperate |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-IQ2_XXS.gguf) | i1-IQ2_XXS | 9.5 |  |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-IQ2_XS.gguf) | i1-IQ2_XS | 10.4 |  |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-IQ2_S.gguf) | i1-IQ2_S | 10.9 |  |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-IQ2_M.gguf) | i1-IQ2_M | 11.7 |  |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-Q2_K.gguf) | i1-Q2_K | 12.9 | IQ3_XXS probably better |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-IQ3_XXS.gguf) | i1-IQ3_XXS | 13.1 | lower quality |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-IQ3_XS.gguf) | i1-IQ3_XS | 14.1 |  |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-Q3_K_S.gguf) | i1-Q3_K_S | 14.8 | IQ3_XS probably better |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-IQ3_S.gguf) | i1-IQ3_S | 14.9 | beats Q3_K* |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-IQ3_M.gguf) | i1-IQ3_M | 15.3 |  |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-Q3_K_M.gguf) | i1-Q3_K_M | 16.3 | IQ3_S probably better |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-Q3_K_L.gguf) | i1-Q3_K_L | 17.7 | IQ3_M probably better |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-IQ4_XS.gguf) | i1-IQ4_XS | 17.9 |  |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-Q4_0.gguf) | i1-Q4_0 | 18.9 | fast, low quality |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-Q4_K_S.gguf) | i1-Q4_K_S | 18.9 | optimal size/speed/quality |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-Q4_K_M.gguf) | i1-Q4_K_M | 19.9 | fast, recommended |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-Q5_K_S.gguf) | i1-Q5_K_S | 22.6 |  |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-Q5_K_M.gguf) | i1-Q5_K_M | 23.2 |  |
+| [GGUF](https://huggingface.co/mradermacher/c4ai-command-r-08-2024-i1-GGUF/resolve/main/c4ai-command-r-08-2024.i1-Q6_K.gguf) | i1-Q6_K | 26.6 | practically like static Q6_K |
+Here is a handy graph by ikawrakow comparing some lower-quality quant
+types (lower is better):
+![image.png](https://www.nethype.de/huggingface_embed/quantpplgraph.png)
+And here are Artefact2's thoughts on the matter:
+https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9
+## FAQ / Model Request
+See https://huggingface.co/mradermacher/model_requests for some answers to
+questions you might have and/or if you want some other model quantized.
+## Thanks
+I thank my company, [nethype GmbH](https://www.nethype.de/), for letting
+me use its servers and providing upgrades to my workstation to enable
+this work in my free time. Additional thanks to [@nicoboss](https://huggingface.co/nicoboss) for giving me access to his private supercomputer, enabling me to provide many more imatrix quants, at much higher quality, than I would otherwise be able to.
+<!-- end -->