metadata
language:
- en
pipeline_tag: image-to-image
tags:
- Diffusion Models
- Stable Diffusion
- ControlNet
- Perturbed-Attention Guidance
- PAG
Super-Resolution with Perturbed-Attention Guidance
This repository is based on Diffusers.
ControlNet is a neural network structure to control diffusion models by adding extra conditions. The pipeline is a modification of StableDiffusionControlNetPipeline to support image generation with ControlNet and Perturbed-Attention Guidance (PAG).
In addition to the examples provided with Openpose, you can also generate using various conditions. Please refer to "ControlNet" section of an official document for details.
Quickstart
Loading ControlNet and Custom Piepline:
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline
controlnet = ControlNetModel.from_pretrained(
"lllyasviel/sd-controlnet-openpose",
torch_dtype=torch.float16
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
custom_pipeline="hyoungwoncho/sd_perturbed_attention_guidance_controlnet",
controlnet=controlnet,
torch_dtype=torch.float16
)
device="cuda"
pipe = pipe.to(device)
Prepare Conditional Images:
from controlnet_aux import OpenposeDetector
openpose = OpenposeDetector.from_pretrained("lllyasviel/ControlNet")
original_image = load_image(
"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/person.png"
)
openpose_image = openpose(original_image)
prompts=""
Conditional Generation with ControlNet and PAG:
output = pipe(
prompts,
image=openpose_image,
num_inference_steps=50,
guidance_scale=0.0,
pag_scale=4.0,
pag_applied_layers_index=["m0"]
).images[0]
Parameters
guidance_scale : gudiance scale of CFG (ex: 7.5)
pag_scale : gudiance scale of PAG (ex: 4.0)
pag_applied_layers_index : index of the layer to apply perturbation (ex: ['m0'])