---
inference: false
license: cc-by-nc-4.0
library_name: transformers
language:
- en
- fr
- de
- es
- it
- pt
- ja
- ko
- zh
- ar
extra_gated_prompt: "By submitting this form, you agree to the [License Agreement](https://cohere.com/c4ai-cc-by-nc-license)  and acknowledge that the information you provide will be collected, used, and shared in accordance with Cohere’s [Privacy Policy]( https://cohere.com/privacy)." 
extra_gated_fields:
 Name: text
 Affiliation: text
 Country:
    type: select
    options: 
      - Aruba
      - Afghanistan
      - Angola
      - Anguilla
      - Åland Islands
      - Albania
      - Andorra
      - United Arab Emirates
      - Argentina
      - Armenia
      - American Samoa
      - Antarctica
      - French Southern Territories
      - Antigua and Barbuda
      - Australia
      - Austria
      - Azerbaijan
      - Burundi
      - Belgium
      - Benin
      - Bonaire Sint Eustatius and Saba
      - Burkina Faso
      - Bangladesh
      - Bulgaria
      - Bahrain
      - Bahamas
      - Bosnia and Herzegovina
      - Saint Barthélemy
      - Belarus
      - Belize
      - Bermuda
      - Plurinational State of Bolivia
      - Brazil
      - Barbados
      - Brunei-Darussalam
      - Bhutan
      - Bouvet-Island
      - Botswana
      - Central African Republic
      - Canada
      - Cocos (Keeling) Islands
      - Switzerland
      - Chile
      - China
      - Côte-dIvoire
      - Cameroon
      - Democratic Republic of the Congo
      - Cook Islands
      - Colombia
      - Comoros
      - Cabo Verde
      - Costa Rica
      - Cuba
      - Curaçao
      - Christmas Island
      - Cayman Islands
      - Cyprus
      - Czechia
      - Germany
      - Djibouti
      - Dominica
      - Denmark
      - Dominican Republic
      - Algeria
      - Ecuador
      - Egypt
      - Eritrea
      - Western Sahara
      - Spain
      - Estonia
      - Ethiopia
      - Finland
      - Fiji
      - Falkland Islands (Malvinas)
      - France
      - Faroe Islands
      - Federated States of Micronesia
      - Gabon
      - United Kingdom
      - Georgia
      - Guernsey
      - Ghana
      - Gibraltar
      - Guinea
      - Guadeloupe
      - Gambia
      - Guinea Bissau
      - Equatorial Guinea
      - Greece
      - Grenada
      - Greenland
      - Guatemala
      - French Guiana
      - Guam
      - Guyana
      - Hong Kong
      - Heard Island and McDonald Islands
      - Honduras
      - Croatia
      - Haiti
      - Hungary
      - Indonesia
      - Isle of Man
      - India
      - British Indian Ocean Territory
      - Ireland
      - Islamic Republic of Iran
      - Iraq
      - Iceland
      - Israel
      - Italy
      - Jamaica
      - Jersey
      - Jordan
      - Japan
      - Kazakhstan
      - Kenya
      - Kyrgyzstan
      - Cambodia
      - Kiribati
      - Saint-Kitts-and-Nevis
      - South Korea
      - Kuwait
      - Lao-Peoples-Democratic-Republic
      - Lebanon
      - Liberia
      - Libya
      - Saint-Lucia
      - Liechtenstein
      - Sri Lanka
      - Lesotho
      - Lithuania
      - Luxembourg
      - Latvia
      - Macao
      - Saint Martin (French-part)
      - Morocco
      - Monaco
      - Republic of Moldova
      - Madagascar
      - Maldives
      - Mexico
      - Marshall Islands
      - North Macedonia
      - Mali
      - Malta
      - Myanmar
      - Montenegro
      - Mongolia
      - Northern Mariana Islands
      - Mozambique
      - Mauritania
      - Montserrat
      - Martinique
      - Mauritius
      - Malawi
      - Malaysia
      - Mayotte
      - Namibia
      - New Caledonia
      - Niger
      - Norfolk Island
      - Nigeria
      - Nicaragua
      - Niue
      - Netherlands
      - Norway
      - Nepal
      - Nauru
      - New Zealand
      - Oman
      - Pakistan
      - Panama
      - Pitcairn
      - Peru
      - Philippines
      - Palau
      - Papua New Guinea
      - Poland
      - Puerto Rico
      - North Korea
      - Portugal
      - Paraguay
      - State of Palestine
      - French Polynesia
      - Qatar
      - Réunion
      - Romania
      - Russia
      - Rwanda
      - Saudi Arabia
      - Sudan
      - Senegal
      - Singapore
      - South Georgia and the South Sandwich Islands
      - Saint Helena Ascension and Tristan da Cunha
      - Svalbard and Jan Mayen
      - Solomon Islands
      - Sierra Leone
      - El Salvador
      - San Marino
      - Somalia
      - Saint Pierre and Miquelon
      - Serbia
      - South Sudan
      - Sao Tome and Principe
      - Suriname
      - Slovakia
      - Slovenia
      - Sweden
      - Eswatini
      - Sint Maarten (Dutch-part)
      - Seychelles
      - Syrian Arab Republic
      - Turks and Caicos Islands
      - Chad
      - Togo
      - Thailand
      - Tajikistan
      - Tokelau
      - Turkmenistan
      - Timor Leste
      - Tonga
      - Trinidad and Tobago
      - Tunisia
      - Turkey
      - Tuvalu
      - Taiwan 
      - United Republic of Tanzania
      - Uganda
      - Ukraine
      - United States Minor Outlying Islands
      - Uruguay
      - United-States
      - Uzbekistan
      - Holy See (Vatican City State)
      - Saint Vincent and the Grenadines
      - Bolivarian Republic of Venezuela
      - Virgin Islands British
      - Virgin Islands U.S.
      - VietNam
      - Vanuatu
      - Wallis and Futuna
      - Samoa
      - Yemen
      - South Africa
      - Zambia
      - Zimbabwe
 Receive email updates on C4AI and Cohere research, events, products and services?:
   type: select
   options: 
     - Yes
     - No
 I agree to use this model for non-commercial use ONLY: checkbox
---

Quantized model => https://huggingface.co/CohereForAI/c4ai-command-r-plus-08-2024

**Quantization Details:**  
Quantization is done using turboderp's ExLlamaV2 v0.2.1. 

I use the default calibration datasets and arguments. The repo also includes a "measurement.json" file, which was used during the quantization process.

For models with bits per weight (BPW) over 6.0, I default to quantizing the `lm_head` layer at 8 bits instead of the standard 6 bits.



---

**Who are you? What's with these weird BPWs on [insert model here]?**  
I specialize in optimized EXL2 quantization for models in the 70B to 100B+ range, specifically tailored for 48GB VRAM setups. My rig is built using 2 x 3090s with a Ryzen APU (APU used solely for desktop output—no VRAM wasted on the 3090s). I use TabbyAPI for inference, targeting context sizes between 32K and 64K.

Every model I upload includes a `config.yml` file with my ideal TabbyAPI settings. If you're using my config, don’t forget to set `PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync` to save some VRAM.