File size: 4,733 Bytes
a48af49 69bb057 a48af49 69bb057 a48af49 69bb057 a48af49 69bb057 1fd8579 a48af49 deb71be a48af49 69bb057 a48af49 69bb057 a48af49 69bb057 a48af49 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 |
---
license: apache-2.0
language:
- en
tags:
- Kolors
- text-to-image
- stable-diffusion
library_name: diffusers
---
# Kolors-ControlNet-Canny weights and inference code
## <a name="Introduction"></a>📖 Introduction
We provide two ControlNet weights and inference code based on Kolors-Basemodel: Canny and Depth. You can find some example images below.
**1、ControlNet Demos**
<img src="demo1.png">
**2、ControlNet and IP-Adapter-Plus Demos**
We also support joint inference code between Kolors-IPadapter and Kolors-ControlNet.
<img src="demo2.png">
## <a name="Evaluation"></a>📊 Evaluation
To evaluate the performance of models, we compiled a test set of more than 200 images and text prompts. We invite several image experts to provide fair ratings for the generated results of different models. The experts rate the generated images based on four criteria: visual appeal, text faithfulness, conditional controllability, and overall satisfaction. Conditional controllability measures controlnet's ability to preserve spatial structure, while the other criteria follow the evaluation standards of BaseModel. The specific results are summarized in the table below, where Kolors-ControlNet achieved better performance in various criterias.
**1、Canny**
| Model | Average Overall Satisfaction | Average Visual Appeal | Average Text Faithfulness | Average Conditional Controllability |
| :--------------: | :--------: | :--------: | :--------: | :--------: |
| SDXL-ControlNet-Canny | 3.14 | 3.63 | 4.37 | 2.84 |
| **Kolors-ControlNet-Canny** | **4.06** | **4.64** | **4.45** | **3.52** |
**2、Depth**
| Model | Average Overall Satisfaction | Average Visual Appeal | Average Text Faithfulness | Average Conditional Controllability |
| :--------------: | :--------: | :--------: | :--------: | :--------: |
| SDXL-ControlNet-Depth | 3.35 | 3.77 | 4.26 | 4.5 |
| **Kolors-ControlNet-Depth** | **4.12** | **4.12** | **4.62** | **4.6** |
<img src="compare_demo.png">
<font color=gray style="font-size:12px">*The [SDXL-ControlNet-Canny](https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0) and [SDXL-ControlNet-Depth](https://huggingface.co/diffusers/controlnet-depth-sdxl-1.0) load [DreamShaper-XL](https://civitai.com/models/112902?modelVersionId=351306) as backbone model.*</font>
------
## <a name="Usage"></a>🛠️ Usage
### Requirements
The dependencies and installation are basically the same as the [Kolors-BaseModel](https://huggingface.co/Kwai-Kolors/Kolors).
### Weights download：
```bash
# Canny - ControlNet
huggingface-cli download --resume-download Kwai-Kolors/Kolors-ControlNet-Canny --local-dir weights/Kolors-ControlNet-Canny
# Depth - ControlNet
huggingface-cli download --resume-download Kwai-Kolors/Kolors-ControlNet-Depth --local-dir weights/Kolors-ControlNet-Depth
```
If you intend to utilize the depth estimation network, please make sure to download its corresponding model weights.
```
huggingface-cli download lllyasviel/Annotators ./dpt_hybrid-midas-501f0c75.pt --local-dir ./controlnet/annotator/ckpts
```
### Inference：
**a. Using canny ControlNet:**
```bash
python ./controlnet/sample_controlNet.py ./controlnet/assets/woman_1.png 一个漂亮的女孩，高品质，超清晰，色彩鲜艳，超高分辨率，最佳品质，8k，高清，4K Canny
python ./controlnet/sample_controlNet.py ./controlnet/assets/dog.png 全景，一只可爱的白色小狗坐在杯子里，看向镜头，动漫风格，3d渲染，辛烷值渲染 Canny
# The image will be saved to "controlnet/outputs/"
```
**b. Using depth ControlNet:**
```bash
python ./controlnet/sample_controlNet.py ./controlnet/assets/woman_2.png 新海诚风格，丰富的色彩，穿着绿色衬衫的女人站在田野里，唯美风景，清新明亮，斑驳的光影，最好的质量，超细节，8K画质 Depth
python ./controlnet/sample_controlNet.py ./controlnet/assets/bird.png 一只颜色鲜艳的小鸟，高品质，超清晰，色彩鲜艳，超高分辨率，最佳品质，8k，高清，4K Depth
# The image will be saved to "controlnet/outputs/"
```
**c. Using depth ControlNet + IP-Adapter-Plus:**
If you intend to utilize the kolors-ip-adapter-plus, please make sure to download its corresponding model weights.
```bash
python ./controlnet/sample_controlNet_ipadapter.py ./controlnet/assets/woman_2.png ./ipadapter/asset/2.png 一个红色头发的女孩，唯美风景，清新明亮，斑驳的光影，最好的质量，超细节，8K画质 Depth
# The image will be saved to "controlnet/outputs/"
```
### Acknowledgments
- Thanks to [ControlNet](https://github.com/lllyasviel/ControlNet) for providing the codebase.
<br>
|