Using LCM allows for a reduction in inference steps. Furthermore, employing LCM-LoRA can reduce the inference steps even more.This time, we will combine LCM-LoRA with the conventional Style-LoRA used in Diffusion Models (DM). LCM-LoRA-SDv1.5（Speedup LCM-LoRA）<a href="https://huggingface.co/latent-consistency/lcm-lora-sdv1-5" rel="noopener noreferrer" target="_blank">https://huggingface.co/latent-consistency/lcm-lora-sdv1-5</a> PixelArtRedmond 1.5V- Pixel Art Loras for SD 1.5!（pixelize Style-LoRA）<a href="https://huggingface.co/artificialguybr/pixelartredmond-1-5v-pixel-art-loras-for-sd-1-5" rel="noopener noreferrer" target="_blank">https://huggingface.co/artificialguybr/pixelartredmond-1-5v-pixel-art-loras-for-sd-1-5</a> <pre class="ql-syntax" spellcheck="false">!pip install -q diffusers accelerate peft
</pre> The base model utilizes the distilled model of Dreamshaper v7 with LCM. FP32 is adopted.<pre class="ql-syntax" spellcheck="false">import torch
from diffusers import LCMScheduler, StableDiffusionPipeline

model_id = &#34;SimianLuo/LCM_Dreamshaper_v7&#34;

pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.to(&#34;cuda&#34;)
</pre> Next, we apply LoRA. This time, both were available on Hugging Face, so we proceed directly.<pre class="ql-syntax" spellcheck="false"># load and fuse lcm lora
pipe.load_lora_weights(&#34;artificialguybr/pixelartredmond-1-5v-pixel-art-loras-for-sd-1-5&#34;)
pipe.load_lora_weights(&#34;latent-consistency/lcm-lora-sdv1-5&#34;)
pipe.fuse_lora()
</pre> <pre class="ql-syntax" spellcheck="false">prompt = &#34;portrait photo of a girl, photograph, highly detailed face, depth of field, moody light, golden hour, style by Dan Winters, Russell James, Steve McCurry, centered, extremely detailed, Nikon D850, award winning photography&#34;
</pre> We&#39;ll try running it with 2 inference steps.<pre class="ql-syntax" spellcheck="false">%%time

# disable guidance_scale by passing 0
image = pipe(prompt=prompt, num_inference_steps=2, guidance_scale=0).images[0]
image
</pre> It&#39;s incredibly fast on the RTX6000ada. It finishes in 0.5 seconds.<img src="https://assets.blueqat.com/public/uploads/us-east-2:4805ff4b-c3cc-4344-b165-86544c34d0bf/2024/03/20/LCM_multilora.png"/><pre class="ql-syntax" spellcheck="false">CPU times: user 752 ms, sys: 102 ms, total: 853 ms
Wall time: 497 ms
</pre> For comparison, I tried inference with LCM-LoRA turned off.<img src="https://assets.blueqat.com/public/uploads/us-east-2:4805ff4b-c3cc-4344-b165-86544c34d0bf/2024/03/20/LCM2.png"/> The style of the image changed significantly. This confirmed that multiple LoRAs were effectively applied. That&#39;s all.

Combining LCM-LoRA + Style-LoRA for speed improvement & style modification. RTX6000ada benchmark.

Yuichiro Minato