LCM, or Latent Consistency Models, is an attempt to reduce the denoising steps in diffusion models significantly by training neural networks (NN) to solving ordinary differential equations (ODEs) for denoising. In this context, a model that further accelerates the process by reducing the denoising steps has emerged, extending the use of LoRA, a low-rank adapter, in the realm of LCM. LCM is sufficiently fast that expensive GPUs are not necessary. Here, we experimented with running LCM and LoRA on a Google Colab T4 GPU. <a href="https://github.com/luosiallen/latent-consistency-model" rel="noopener noreferrer" target="_blank">https://github.com/luosiallen/latent-consistency-model</a> <pre class="ql-syntax" spellcheck="false">!pip install -q diffusers accelerate peft
</pre> This time, either fp16 or fp32 is fine, but we&#39;ll opt for 16 due to its speed.We will use the weights from LCM.<pre class="ql-syntax" spellcheck="false">from diffusers import StableDiffusionPipeline, LCMScheduler
import torch

pipe = StableDiffusionPipeline.from_pretrained(&#34;SimianLuo/LCM_Dreamshaper_v7&#34;, torch_dtype=torch.float16).to(&#34;cuda&#34;)
</pre> We will use the prompt provided in the example.<pre class="ql-syntax" spellcheck="false">prompt = &#34;portrait photo of a girl, photograph, highly detailed face, depth of field, moody light, golden hour, style by Dan Winters, Russell James, Steve McCurry, centered, extremely detailed, Nikon D850, award winning photography&#34;
</pre> First, let&#39;s look at the one without using LoRA. We&#39;ll examine steps 2, 4, and 8.<pre class="ql-syntax" spellcheck="false">%%time

image = pipe(prompt=prompt, num_inference_steps=2, guidance_scale=8.0, lcm_origin_steps=50).images[0]

image.save(&#34;image.png&#34;)
image
</pre> LCM without LoRA / steps 2 / 768x768 / 1.48s<img src="https://assets.blueqat.com/public/uploads/us-east-2:4805ff4b-c3cc-4344-b165-86544c34d0bf/2024/03/20/lcm2.png"/> LCM without LoRA / steps 4 / 768x768 / 1.94s<img src="https://assets.blueqat.com/public/uploads/us-east-2:4805ff4b-c3cc-4344-b165-86544c34d0bf/2024/03/20/lcm4.png"/> LCM without LoRA / steps 8 / 768x768 / 2.94s<img src="https://assets.blueqat.com/public/uploads/us-east-2:4805ff4b-c3cc-4344-b165-86544c34d0bf/2024/03/20/lcm8.png"/> Next, we will attach LCM-LoRA.While LoRA typically modifies the style, this version reduces the inference steps. <a href="https://huggingface.co/latent-consistency/lcm-lora-sdv1-5" rel="noopener noreferrer" target="_blank">https://huggingface.co/latent-consistency/lcm-lora-sdv1-5</a>The guidance scale is suggested to be set to 0, or choose between 1 to 2, so we will set it to 0. <pre class="ql-syntax" spellcheck="false">pipe.load_lora_weights(&#34;latent-consistency/lcm-lora-sdv1-5&#34;)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
</pre> LCM with LCM-LoRA / steps 2 / 768x768 / 1.41s<img src="https://assets.blueqat.com/public/uploads/us-east-2:4805ff4b-c3cc-4344-b165-86544c34d0bf/2024/03/20/lora2.png"/> The style has changed slightly. Even with just 2 steps, the image emerged without being too blurry. LCM with LCM-LoRA / steps 4 / 768x768 / 2.02s<img src="https://assets.blueqat.com/public/uploads/us-east-2:4805ff4b-c3cc-4344-b165-86544c34d0bf/2024/03/20/lora4.png"/> LCM with LCM-LoRA / steps 8 / 768x768 / 3.13s<img src="https://assets.blueqat.com/public/uploads/us-east-2:4805ff4b-c3cc-4344-b165-86544c34d0bf/2024/03/20/lora8.png"/> It&#39;s great that a proper image comes out even with just 2 steps.

LCM / Latent Consistency Models and LCM-LoRA benchmark

Yuichiro Minato