kvaishnavi commited on
Commit
0f87da3
β€’
1 Parent(s): 3d29cbc

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +91 -94
README.md CHANGED
@@ -1,94 +1,91 @@
1
- ---
2
- license: mit
3
- tags:
4
- - ONNX
5
- - DML
6
- - ONNXRuntime
7
- - phi3
8
- - custom_code
9
- ---
10
-
11
- # Phi-3 Vision-128k-Instruct ONNX CPU models
12
-
13
- <!-- Provide a quick summary of what the model is/does. -->
14
- This repository hosts the optimized versions of [Phi-3-vision-128k-instruct](https://aka.ms/phi3-vision-128k-instruct) to accelerate inference with ONNX Runtime for your CPU.
15
-
16
- Phi-3 Vision is a lightweight, state-of-the-art open multimodal model built upon datasets that include synthetic data and filtered publicly available web data with a focus on very high-quality, reasoning dense data both on text and vision. The model belongs to the Phi-3 model family, and the multimodal version supports up to 128K context length (in tokens). The base model has undergone a rigorous enhancement process, incorporating both supervised fine-tuning and direct preference optimization, to ensure precise instruction adherence and robust safety measures.
17
-
18
- Optimized variants of the Phi-3 Vision models are published here in [ONNX](https://onnx.ai) format to run with [ONNX Runtime](https://onnxruntime.ai/) on CPU and GPU across devices, including server platforms, Windows, Linux and Mac desktops, and mobile CPUs, with the precision best suited to each of these targets.
19
-
20
- ## ONNX Models
21
-
22
- Here are some of the optimized configurations we have added:
23
-
24
- 1. ONNX model for INT4 CPU: ONNX model for CPUs using int4 quantization via RTN.
25
-
26
- How do you know which is the best ONNX model for you:
27
- - Are you on a Windows machine with GPU?
28
- - I don't know β†’ Review this [guide](https://www.microsoft.com/en-us/windows/learning-center/how-to-check-gpu) to see whether you have a GPU in your Windows machine.
29
- - Yes β†’ Access the Hugging Face DirectML ONNX models and instructions at [Phi-3-vision-128k-instruct-onnx-directml (coming soon)](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct-onnx-directml).
30
- - No β†’ Do you have a NVIDIA GPU?
31
- - I don't know β†’ Review this [guide](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html#verify-you-have-a-cuda-capable-gpu) to see whether you have a CUDA-capable GPU.
32
- - Yes β†’ Access the Hugging Face CUDA ONNX models and instructions at [Phi-3-vision-128k-instruct-onnx-cuda](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct-onnx-cuda) for NVIDIA GPUs.
33
- - No β†’ Access the Hugging Face ONNX models for CPU devices and instructions at [Phi-3-vision-128k-instruct-onnx-cpu](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct-onnx-cpu)
34
-
35
- ## How to Get Started with the Model
36
- To support the Phi-3 models across a range of devices, platforms, and EP backends, we introduce a new API to wrap several aspects of generative AI inferencing. This API makes it easy to drag and drop LLMs straight into your app. To run the early version of these models with ONNX, follow the steps [here](http://aka.ms/generate-tutorial). You can also test this with a [chat app](https://github.com/microsoft/onnxruntime-genai/tree/main/examples/chat_app).
37
-
38
- ## Hardware Supported
39
-
40
- The models are tested on:
41
- - Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz
42
-
43
- Minimum Configuration Required:
44
- - CPU machine with 16GB RAM
45
-
46
- ### Model Description
47
-
48
- - **Developed by:** Microsoft
49
- - **Model type:** ONNX
50
- - **Language(s) (NLP):** Python, C, C++
51
- - **License:** MIT
52
- - **Model Description:** This is a conversion of the Phi-3 Vision-128K-Instruct model for ONNX Runtime inference.
53
-
54
- ## Additional Details
55
- - [**Phi-3 Small, Medium, and Vision Blog**](https://aka.ms/phi3_ONNXBuild24) and [**Phi-3 Mini Blog**](https://aka.ms/phi3-optimizations)
56
- - [**Phi-3 Model Blog Link**](https://aka.ms/phi3blog-april)
57
- - [**Phi-3 Model Card**](https://aka.ms/phi3-vision-128k-instruct)
58
- - [**Phi-3 Technical Report**](https://aka.ms/phi3-tech-report)
59
- - [**Phi-3 on Azure AI Studio**](https://aka.ms/phi3-azure-ai)
60
-
61
- ## Performance Metrics
62
- The performance of the ONNX vision model is similar to [Phi-3-mini-128k-instruct-onnx](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct-onnx) during token generation.
63
-
64
- ## Base Model Usage and Considerations
65
- **Primary use cases**
66
-
67
- The model is intended for broad commercial and research use in English. The model provides uses for general purpose AI systems and applications with visual and text input capabilities which require
68
-
69
- 1) memory/compute constrained environments;
70
- 2) latency bound scenarios;
71
- 3) general image understanding;
72
- 4) OCR;
73
- 5) chart and table understanding.
74
-
75
- Our model is designed to accelerate research on efficient language and multimodal models, for use as a building block for generative AI powered features.
76
-
77
- **Use case considerations**
78
-
79
- Our models are not specifically designed or evaluated for all downstream purposes. Developers should consider common limitations of language models as they select use cases, and evaluate and mitigate for accuracy, safety, and fairness before using within a specific downstream use case, particularly for high-risk scenarios.
80
-
81
- Developers should be aware of and adhere to applicable laws or regulations (including privacy, trade compliance laws, etc.) that are relevant to their use case.
82
-
83
- Nothing contained in this Model Card should be interpreted as or deemed a restriction or modification to the license the model is released under.
84
-
85
- ## Appendix
86
-
87
- ### Activation Aware Quantization
88
- AWQ works by identifying the top 1% most salient weights that are most important for maintaining accuracy and quantizing the remaining 99% of weights. This leads to less accuracy loss from quantization compared to many other quantization techniques. For more on AWQ see [here](https://arxiv.org/abs/2306.00978).
89
-
90
- ## Model Card Contact
91
- parinitarahi, kvaishnavi, natke
92
-
93
- ## Contributors
94
- Kunal Vaishnavi, Sunghoon Choi, Yufeng Li, Baiju Meswani, Sheetal Arun Kadam, Rui Ren, Natalie Kershaw, Parinita Rahi
 
1
+ ---
2
+ license: mit
3
+ tags:
4
+ - ONNX
5
+ - DML
6
+ - ONNXRuntime
7
+ - phi3
8
+ - custom_code
9
+ ---
10
+
11
+ # Phi-3 Vision-128k-Instruct ONNX CPU models
12
+
13
+ <!-- Provide a quick summary of what the model is/does. -->
14
+ This repository hosts the optimized versions of [Phi-3-vision-128k-instruct](https://aka.ms/phi3-vision-128k-instruct) to accelerate inference with ONNX Runtime for your CPU.
15
+
16
+ Phi-3 Vision is a lightweight, state-of-the-art open multimodal model built upon datasets that include synthetic data and filtered publicly available web data with a focus on very high-quality, reasoning dense data both on text and vision. The model belongs to the Phi-3 model family, and the multimodal version supports up to 128K context length (in tokens). The base model has undergone a rigorous enhancement process, incorporating both supervised fine-tuning and direct preference optimization, to ensure precise instruction adherence and robust safety measures.
17
+
18
+ Optimized variants of the Phi-3 Vision models are published here in [ONNX](https://onnx.ai) format to run with [ONNX Runtime](https://onnxruntime.ai/) on CPU and GPU across devices, including server platforms, Windows, Linux and Mac desktops, and mobile CPUs, with the precision best suited to each of these targets.
19
+
20
+ ## ONNX Models
21
+
22
+ Here are some of the optimized configurations we have added:
23
+
24
+ 1. ONNX model for INT4 CPU: ONNX model for CPUs using int4 quantization via RTN.
25
+
26
+ How do you know which is the best ONNX model for you:
27
+ - Are you on a Windows machine with GPU?
28
+ - I don't know β†’ Review this [guide](https://www.microsoft.com/en-us/windows/learning-center/how-to-check-gpu) to see whether you have a GPU in your Windows machine.
29
+ - Yes β†’ Access the Hugging Face DirectML ONNX models and instructions at [Phi-3-vision-128k-instruct-onnx-directml (coming soon)](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct-onnx-directml).
30
+ - No β†’ Do you have a NVIDIA GPU?
31
+ - I don't know β†’ Review this [guide](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html#verify-you-have-a-cuda-capable-gpu) to see whether you have a CUDA-capable GPU.
32
+ - Yes β†’ Access the Hugging Face CUDA ONNX models and instructions at [Phi-3-vision-128k-instruct-onnx-cuda](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct-onnx-cuda) for NVIDIA GPUs.
33
+ - No β†’ Access the Hugging Face ONNX models for CPU devices and instructions at [Phi-3-vision-128k-instruct-onnx-cpu](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct-onnx-cpu)
34
+
35
+ ## How to Get Started with the Model
36
+ To support the Phi-3 models across a range of devices, platforms, and EP backends, we introduce a new API to wrap several aspects of generative AI inferencing. This API makes it easy to drag and drop LLMs straight into your app. To run the early version of these models with ONNX, follow the steps [here](http://aka.ms/generate-tutorial). You can also test this with a [chat app](https://github.com/microsoft/onnxruntime-genai/tree/main/examples/chat_app).
37
+
38
+ ## Hardware Supported
39
+
40
+ The models are tested on:
41
+ - Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz
42
+
43
+ Minimum Configuration Required:
44
+ - CPU machine with 16GB RAM
45
+
46
+ ### Model Description
47
+
48
+ - **Developed by:** Microsoft
49
+ - **Model type:** ONNX
50
+ - **Language(s) (NLP):** Python, C, C++
51
+ - **License:** MIT
52
+ - **Model Description:** This is a conversion of the Phi-3 Vision-128K-Instruct model for ONNX Runtime inference.
53
+
54
+ ## Additional Details
55
+ - [**Phi-3 Small, Medium, and Vision Blog**](https://aka.ms/phi3_ONNXBuild24) and [**Phi-3 Mini Blog**](https://aka.ms/phi3-optimizations)
56
+ - [**Phi-3 Model Blog Link**](https://aka.ms/phi3blog-april)
57
+ - [**Phi-3 Model Card**](https://aka.ms/phi3-vision-128k-instruct)
58
+ - [**Phi-3 Technical Report**](https://aka.ms/phi3-tech-report)
59
+ - [**Phi-3 on Azure AI Studio**](https://aka.ms/phi3-azure-ai)
60
+
61
+ ## Performance Metrics
62
+ The performance of the ONNX vision model is similar to [Phi-3-mini-128k-instruct-onnx](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct-onnx) during token generation.
63
+
64
+ ## Base Model Usage and Considerations
65
+ **Primary use cases**
66
+
67
+ The model is intended for broad commercial and research use in English. The model provides uses for general purpose AI systems and applications with visual and text input capabilities which require
68
+
69
+ 1) memory/compute constrained environments;
70
+ 2) latency bound scenarios;
71
+ 3) general image understanding;
72
+ 4) OCR;
73
+ 5) chart and table understanding.
74
+
75
+ Our model is designed to accelerate research on efficient language and multimodal models, for use as a building block for generative AI powered features.
76
+
77
+ **Use case considerations**
78
+
79
+ Our models are not specifically designed or evaluated for all downstream purposes. Developers should consider common limitations of language models as they select use cases, and evaluate and mitigate for accuracy, safety, and fairness before using within a specific downstream use case, particularly for high-risk scenarios.
80
+
81
+ Developers should be aware of and adhere to applicable laws or regulations (including privacy, trade compliance laws, etc.) that are relevant to their use case.
82
+
83
+ Nothing contained in this Model Card should be interpreted as or deemed a restriction or modification to the license the model is released under.
84
+
85
+ ## Appendix
86
+
87
+ ## Model Card Contact
88
+ parinitarahi, kvaishnavi, natke
89
+
90
+ ## Contributors
91
+ Kunal Vaishnavi, Sunghoon Choi, Yufeng Li, Baiju Meswani, Sheetal Arun Kadam, Rui Ren, Natalie Kershaw, Parinita Rahi