Hugging Face's logo

kohya-ss
/
controlnet-lllite

Model card Files Community
4
controlnet-lllite
File size: 3,716 Bytes 
1a31faf
 
 
460f3e5
3486b9d
460f3e5
 
 
f6198a1
 
3486b9d
 
f6198a1
 
2ae37a3
 
3486b9d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
937e7b9
 
e2b9ae8
 
 
 
 
 
3486b9d
e2b9ae8
 
3486b9d
 
 
 
 
 
 
 
 
 
 
 
 
460f3e5
 
 
 
 
 
 
d4883d6
2de0a69
 
 
 
 
 
 
 
6f0cf8f
 
 
 
 
 
 
 
a1614ae
 
 
 
 
 
 
 
6b01a8a
 
 
 
 
 
 
 
e2b9ae8
 
 
937e7b9
e2b9ae8
937e7b9
d4883d6
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
 
---
license: apache-2.0
---

Pre-trained models and output samples of ControlNet-LLLite.

Note: The model structure is highly experimental and may be subject to change in the future.

Inference with ComfyUI: https://github.com/kohya-ss/ControlNet-LLLite-ComfyUI

For 1111's Web UI, [sd-webui-controlnet](https://github.com/Mikubill/sd-webui-controlnet) extension supports ControlNet-LLLite.

Training: https://github.com/kohya-ss/sd-scripts/blob/sdxl/docs/train_lllite_README.md

The recommended preprocessing for the blur model is Gaussian blur.

# Naming Rules

`controllllite_v01032064e_sdxl_blur_500-1000.safetensors`

- `v01` : Version Flag.
- `032` : Dimensions of conditioning.
- `064` : Dimensions of control module.
- `sdxl` : Base Model.
- `blur` : The control method. `anime` means the LLLite model is trained on/with anime sdxl model and images.
- `500-1000` : (Optional) Timesteps for training. If this is `500-1000`, please control only the first half step.

# Models

## Trained on sdxl base

- controllllite_v01032064e_sdxl_blur-500-1000.safetensors
  - trained with 3,919 generated images and Gaussian blur preprocessing.
- controllllite_v01032064e_sdxl_canny.safetensors
  - trained with 3,919 generated images and canny preprocessing.
- controllllite_v01032064e_sdxl_depth_500-1000.safetensors
  - trained with 3,919 generated images and MiDaS v3 - Large preprocessing.

## Trained on anime model

The model ControlNet trained on is our custom model.

- controllllite_v01016032e_sdxl_blur_anime_beta.safetensors
  - beta version.
- controllllite_v01032064e_sdxl_blur-anime_500-1000.safetensors
  - trained with 2,836 generated images and Gaussian blur preprocessing.
- controllllite_v01032064e_sdxl_canny_anime.safetensors
  - trained with 921 generated images and canny preprocessing.
- controllllite_v01008016e_sdxl_depth_anime.safetensors
  - trained with 1,433 generated images and MiDaS v3 - Large preprocessing.
- controllllite_v01032064e_sdxl_fake_scribble_anime.safetensors
  - trained with 921 generated images and PiDiNet preprocessing.
- controllllite_v01032064e_sdxl_pose_anime.safetensors
  - trained with 921 generated images and MMPose preprocessing.
- controllllite_v01032064e_sdxl_pose_anime_v2_500-1000.safetensors
  - trained with 1,415 generated images and MMPose preprocessing.
- controllllite_v01016032e_sdxl_replicate_anime_0-500.safetensors
  - trained with 896 generated image pairs, 1024x1024 and 2048x2048 (highres. fix-ed).
  - Trained for 0-500 steps, but it seems to work for 0-1000.
- controllllite_v01032064e_sdxl_replicate_anime_v2.safetensors
  - trained with 896 generated image pairs, 1024x1024 and 2048x2048 (highres. fix-ed).
  - Trained for 0-1000 steps.

## About replicate model

- Replicates the control image, mixed with the prompt, as possible as the model can.
- No preprocessor is required. Also works for img2img.

# Samples

## sdxl base

![blur](./bs.jpg)

![canny](./cs.jpg)

![depth](./ds.jpg)

## anime model

![source 1](./canny1.png)

![sample 1](./sample1.jpg)

![source 2](./canny2.png)

![sample 2](./sample2.jpg)

![source 3](./depth1.png)

![sample 3](./sample3.jpg)

![source 4](./depth2.png)

![sample 4](./sample4.jpg)

![source 5](./scribble1.png)

![sample 5](./sample5.jpg)

![source 6](./scribble2.png)

![sample 6](./sample6.jpg)

![source 7](./pose1.png)

![sample 7](./sample7.jpg)

![source 8](./pose2.png)

![sample 8](./sample8.jpg)

![source 9](./blur1.jpg)

![sample 9](./sample9.jpg)

![source 10](./blur2.jpg)

![sample 10](./sample10.jpg)

### replicate

![replicate anime_sample 1](./replicate_anime_sample1.jpg)

![replicate anime_sample 2](./replicate_anime_sample2.jpg)

Sample images are generated by custom model.