|
--- |
|
--- |
|
license: creativeml-openrail-m |
|
tags: |
|
- stable-diffusion |
|
- text-to-image |
|
datasets: |
|
- ProGamerGov/StableDiffusion-v1-5-Regularization-Images |
|
--- |
|
# Ukeiyo-style Diffusion |
|
|
|
This is the fine-tuned Stable Diffusion model trained on traditional Japanese Ukeiyo-style images. |
|
Use the tokens **_ukeiyoddim style_** in your prompts for the effect. |
|
The model repo also contains a ckpt file , so that you can use the model with your own implementation of |
|
stable diffusion. |
|
|
|
### 🧨 Diffusers |
|
|
|
This model can be used just like any other Stable Diffusion model. For more information, |
|
please have a look at the [Stable Diffusion](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion). |
|
|
|
You can also export the model to [ONNX](https://huggingface.co/docs/diffusers/optimization/onnx), [MPS](https://huggingface.co/docs/diffusers/optimization/mps) and/or [FLAX/JAX](). |
|
|
|
```python |
|
#!pip install diffusers transformers scipy torch |
|
from diffusers import StableDiffusionPipeline |
|
import torch |
|
model_id = "salmonhumorous/ukeiyo-style-diffusion" |
|
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16) |
|
pipe = pipe.to("cuda") |
|
prompt = "illustration of ukeiyoddim style landscape" |
|
image = pipe(prompt).images[0] |
|
image.save("./ukeiyo_landscape.png") |
|
``` |
|
|
|
## Training procedure and data |
|
|
|
The training for this model was done using a RTX 3090. The training was completed in 28 minutes for a total of 2000 steps. A total of 33 instance images (Images of the style I was aiming for) and 1k Regularization images was used. Regularization images dataset used by [ProGamerGov](https://huggingface.co/datasets/ProGamerGov/StableDiffusion-v1-5-Regularization-Images). |
|
|
|
Training notebook used by [Shivam Shrirao](https://colab.research.google.com/github/ShivamShrirao/diffusers/blob/main/examples/dreambooth/DreamBooth_Stable_Diffusion.ipynb). |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- number of steps : 2000 |
|
- learning_rate: 1e-6 |
|
- train_batch_size: 1 |
|
- scheduler_type: DDIM |
|
- number of instance images : 33 |
|
- number of regularization images : 1000 |
|
- lr_scheduler : constant |
|
- gradient_checkpointing |
|
|
|
### Results |
|
|
|
Below are the sample results for different training steps : |
|
 |
|
|
|
### Sample images by model trained for 2000 steps : |
|
|
|
prompt = "landscape" |
|
 |
|
prompt = "ukeiyoddim style landscape" |
|
 |
|
prompt = " illustration of ukeiyoddim style landscape" |
|
 |
|
|
|
 |
|
|
|
### Acknowledgement |
|
|
|
Many thanks to [nitrosocke](https://huggingface.co/nitrosocke), for inspiration and for the [guide](https://github.com/nitrosocke/dreambooth-training-guide). Also thanks, to all the amazing people making stable diffusion easily accessible for everyone. |
|
|
|
|
