FP8 LLMs for vLLM - a neuralmagic Collection

neuralmagic 's Collections

Vision Language Models Quantization

FP8 LLMs for vLLM

Llama-3.2 Quantization

Llama-3.1 Quantization

INT8 LLMs for vLLM

INT4 LLMs for vLLM

Sparse Foundational Llama 2 Models

Compression Papers

DeepSparse Sparse LLMs

Sparse Finetuning MPT

Compressed LLMs from the Community

FP8 LLMs for vLLM

updated 24 days ago

Accurate FP8 quantized models by Neural Magic, ready for use with vLLM!

neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8

Text Generation • Updated Oct 9 • 1.93k • 31
neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8

Text Generation • Updated Oct 9 • 125k • 32
neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8

Text Generation • Updated Oct 9 • 50.9k • 32
neuralmagic/Phi-3-medium-128k-instruct-FP8

Text Generation • Updated Oct 9 • 875 • 5
neuralmagic/Mistral-Nemo-Instruct-2407-FP8

Text Generation • Updated Jul 19 • 29.4k • 16
neuralmagic/Meta-Llama-3-8B-Instruct-FP8

Text Generation • Updated Jul 18 • 14.8k • 18
neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8-dynamic

Text Generation • Updated 22 days ago • 386 • 14
neuralmagic/Meta-Llama-3-70B-Instruct-FP8

Text Generation • Updated Jul 18 • 53.6k • 10
neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8

Text Generation • Updated Jul 18 • 8.34k • 3
neuralmagic/Meta-Llama-3-8B-Instruct-FP8-KV

Text Generation • Updated Jun 19 • 15.3k • 6
neuralmagic/Meta-Llama-3-70B-Instruct-FP8-KV

Text Generation • Updated Jun 26 • 803 • 2
neuralmagic/Mixtral-8x22B-Instruct-v0.1-FP8

Text Generation • Updated Aug 12 • 942 • 2
neuralmagic/Qwen2-72B-Instruct-FP8

Text Generation • Updated Jul 18 • 555 • 10
neuralmagic/Qwen2-7B-Instruct-FP8

Text Generation • Updated Jul 18 • 1.33k • 1
neuralmagic/Qwen2-1.5B-Instruct-FP8

Text Generation • Updated Jul 18 • 33
neuralmagic/Qwen2-0.5B-Instruct-FP8

Text Generation • Updated Jul 18 • 1.09k • 2
neuralmagic/Mistral-7B-Instruct-v0.3-FP8

Text Generation • Updated Jul 18 • 1.88k • 2
neuralmagic/Llama-2-7b-chat-hf-FP8

Text Generation • Updated Jul 18 • 657
neuralmagic/Phi-3-mini-128k-instruct-FP8

Text Generation • Updated Oct 9 • 444
neuralmagic/gemma-2-9b-it-FP8

Text Generation • Updated Jul 18 • 4.45k • 5
neuralmagic/Qwen2-57B-A14B-Instruct-FP8

Text Generation • Updated Jul 18 • 687 • 1
neuralmagic/DeepSeek-Coder-V2-Lite-Instruct-FP8

Text Generation • Updated Jul 18 • 9.39k • 6
neuralmagic/DeepSeek-Coder-V2-Lite-Base-FP8

Text Generation • Updated Jul 18 • 113
neuralmagic/DeepSeek-Coder-V2-Base-FP8

Text Generation • Updated Jul 22 • 14
neuralmagic/DeepSeek-Coder-V2-Instruct-FP8

Text Generation • Updated Jul 22 • 6.84k • 6
neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic

Text Generation • Updated 22 days ago • 3.01k • 5
neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8-dynamic

Text Generation • Updated 22 days ago • 1.93k • 4
neuralmagic/Meta-Llama-3.1-8B-FP8

Text Generation • Updated Oct 9 • 3.53k • 5
neuralmagic/Meta-Llama-3.1-70B-FP8

Text Generation • Updated Oct 9 • 607
neuralmagic/starcoder2-15b-FP8

Text Generation • Updated Oct 9 • 493
neuralmagic/starcoder2-3b-FP8

Text Generation • Updated Oct 9 • 28
neuralmagic/starcoder2-7b-FP8

Text Generation • Updated Oct 9 • 21
neuralmagic/Meta-Llama-3.1-405B-FP8

Text Generation • Updated Oct 9 • 38
neuralmagic/gemma-2-2b-it-FP8

Updated Oct 9 • 347 • 1
neuralmagic/Llama-3.2-1B-Instruct-FP8-dynamic

Text Generation • Updated Oct 9 • 1.29k • 2
neuralmagic/Llama-3.2-3B-Instruct-FP8-dynamic

Text Generation • Updated Oct 9 • 1.79k • 2
neuralmagic/Llama-3.2-3B-Instruct-FP8

Text Generation • Updated 25 days ago • 8.96k • 2
neuralmagic/Llama-3.2-1B-Instruct-FP8

Text Generation • Updated 25 days ago • 236k • 1
neuralmagic/Llama-3.2-1B-FP8

Updated Oct 9 • 66
neuralmagic/Phi-3.5-mini-instruct-FP8-KV

Text Generation • Updated Oct 1 • 270 • 1
neuralmagic/Llama-3.1-Nemotron-70B-Instruct-HF-FP8-dynamic

Text Generation • Updated 24 days ago • 11.2k • 9