Hugging Face's logo

Kooten
/
Noromaid-v0.1-mixtral-8x7b-v3-4bpw-exl2

Text Generation
Transformers
PyTorch
mixtral
text-generation-inference
Inference Endpoints
Model card Files Community
Kooten commited on
Commit
c73869d
•
1 Parent(s): 311bdea

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -5,7 +5,7 @@ license: cc-by-nc-4.0
5
  # Noromaid-v0.1-mixtral-8x7b-v3 4bpw
6
  Exllama quant of [NeverSleep/Noromaid-v0.1-mixtral-8x7b-v3](https://huggingface.co/NeverSleep/Noromaid-v0.1-mixtral-8x7b-v3)
7
 
8
- You probably want to use the [3.5bpw](https://huggingface.co/Kooten/Noromaid-v0.1-mixtral-8x7b-v3-3.5bpw-exl2) version, it lands very close to 24gb of vram with 8k context and 8bit cache enabled. GGUF is probably preferable if you have less than 24gb.
9
 
10
  Other BPW's [2.7bpw](https://huggingface.co/Kooten/Noromaid-v0.1-mixtral-8x7b-v3-2.7bpw-exl2), [3.0bpw](https://huggingface.co/Kooten/Noromaid-v0.1-mixtral-8x7b-v3-3bpw-exl2), [3.5bpw](https://huggingface.co/Kooten/Noromaid-v0.1-mixtral-8x7b-v3-3.5bpw-exl2), [4.0bpw](https://huggingface.co/Kooten/Noromaid-v0.1-mixtral-8x7b-v3-4bpw-exl2)
11
 
 
5
  # Noromaid-v0.1-mixtral-8x7b-v3 4bpw
6
  Exllama quant of [NeverSleep/Noromaid-v0.1-mixtral-8x7b-v3](https://huggingface.co/NeverSleep/Noromaid-v0.1-mixtral-8x7b-v3)
7
 
8
+ You probably want to use the [3.5bpw](https://huggingface.co/Kooten/Noromaid-v0.1-mixtral-8x7b-v3-3.5bpw-exl2) version, it lands very close to 24gb of vram with 8bit cache enabled. GGUF is probably preferable if you have less than 24gb.
9
 
10
  Other BPW's [2.7bpw](https://huggingface.co/Kooten/Noromaid-v0.1-mixtral-8x7b-v3-2.7bpw-exl2), [3.0bpw](https://huggingface.co/Kooten/Noromaid-v0.1-mixtral-8x7b-v3-3bpw-exl2), [3.5bpw](https://huggingface.co/Kooten/Noromaid-v0.1-mixtral-8x7b-v3-3.5bpw-exl2), [4.0bpw](https://huggingface.co/Kooten/Noromaid-v0.1-mixtral-8x7b-v3-4bpw-exl2)
11