Stable Diffusion - Your Ultimate Guide

With all of the talk lately about Artificial Intelligence or AI for short, chances are you will be familiar with some of the more popular AI image generators currently on the market. 

Over the last few years, we’ve seen AI become incredibly advanced, with more and more people tapping into the many benefits it can offer. Now, don’t get us wrong, AI is far from perfect, and it is certainly not without its limitations, but if used correctly, many AI tools out there can certainly be very useful. 

One of the newest AI image generators to hit the market as of late is one known as ‘Stable Diffusion’ and it is this very generator that we are going to be looking at today. This text-to-image, deep learning model may be relatively new (it was only released in 2022) but it is already turning heads and generating quite the buzz, for all the right reasons. 

But what exactly is Stable Diffusion, how does it work, what are its features, and what sets it apart from the other text-to-image generators currently available?

What is Stable Diffusion?

Stable Diffusion is one of the latest generative AI models to hit the market, and despite still being in its infancy, it is already turning heads for all the right reasons. 

Stable Diffusion is a text-to-image model that can produce unique photorealistic images, after receiving both image and text prompts from users. The model was released in 2022 and is based upon diffusion techniques (hence the name). 

As well as producing unique images, Stable Diffusion is also able to perform other secondary functions as well. For example, it is able to produce image-to-image translations via a text prompt, it can perform outpainting, and it can perform inpainting as well, allowing users to touch up and adjust images. Users can also use it to create animations and videos. 

Utilizing latent space, the model needs far fewer processing requirements and can be run on laptops and desktops which feature GPUs. 

Stable Diffusion does not hold any rights to generated images, and the source code is also available to everybody. Users can get as creative as they like and can browse from, and alter, a wide range of generative models and make them their own. 

What are Models in Stable Diffusion?

As mentioned, Stable Diffusion utilizes a form of diffusion model known as a Latent Diffusion Model, or LDM for short. 

When used as part of machine learning, diffusion models are a type of generative models which have the objective of learning a diffusion process which is able to generate the probability distribution of the given dataset. 

How to Use Stable Diffusion Models Effectively

Stable Diffusion is a wonderful tool when it comes to image generation, but it does take a while to get used to. If you’re looking for a way of getting more from your Stable Diffusion sessions, there are things that you need to familiarize yourself with first. 

Negative Prompt 

For those who wish to give Stable Diffusion a bit of a helping hand to ensure it gives them the perfect image that they’re after, Negative Prompt is something they will need to get to grips with. Thankfully, it’s not difficult. 

Negative Prompts allow you to generate images with a simple text input. 

Say, for example, you want to generate an image of a woman with red hair, you may use a positive text prompt, such as ‘portrait of a woman with red hair’. If you decide she shouldn’t have red hair, however, you may use the prompt ‘portrait of a woman without red hair’. 

After this prompt, you may find that the tool generates images of women with even brighter red hair. That’s because SD understood your prompts as ‘woman’ and ‘red hair’. This is where you need to use negative prompts. You would therefore use the prompt ‘portrait of a woman’ followed by the negative prompt ‘red hair’. Now you will generate images of women without red hair. 

Sampling Steps 

Sampling is used by SD to generate an image. 

SD first generates an image, at random, in the latent space. SD’s noise predictor will then roughly estimate the noise in that image. The noise is then taken away from the image. The process is then carried out over and over again a dozen times. The end result is a perfectly clear image. 

This denoising process is referred to as ‘sampling’. This is because SD will generate a new sample image during each step of the denoising process. 

LoRa 

Up next we have LoRa models. 

LoRa (Low-Rank Adaption) models are much smaller than regular checkpoint models. They are small models which apply very small changes to regular checkpoint models. If you have a vast collection of checkpoint models, LoRa models are very appealing because of their size. 

LoRa is used for fine-tuning your Stable Diffusion images. It is essentially a training technique. It works by applying miniscule changes to the most vital parts of Stable Diffusion models. These are known as the cross-attention layers. It is important because it is here where the prompt and the image come together and meet. 

LoRa models play a key role in training the Stable Diffusion tool. They help to provide a finely tuned image as accurately as possible for what the user has requested.

Seed 

Finally, we have Seed. 

Seed is a number used in SD to initialize the generation of an image. It is randomly generated unless specified, yet if you control the seed you will be able to play around with various other parameters,  produce a new of reproducible images, and experiment with prompt variations.

What’s important about Seed is the fact that any image generation which contains the same parameters, seed, and prompts will generate identical images. This is important because it means you can generate a variety of different images based on that initial image. You can then play around with the image and alter it as necessary. You could, for example, generate an image of the same person, but have them expressing different emotions with different facial expressions. 

Text-to-image in Stable Diffusion

One of Stable Diffusion’s most commonly used features is its text-to-image generation capabilities. But what exactly is that? 

Well, text-to-image in Stable Diffusion is basically how it functions. You write a text prompt telling it which image to create, and it then does the rest. 

Users can create different effects and images by altering the denoising schedule, or they can adjust the seed number for a random generator.

Stable Diffusion can generate an image from scratch, simply by using a text prompt where users can describe which elements to include, and which elements to omit. 

As if that wasn’t all, users can also add new elements to the existing image with the use of a text prompt. For example, you could write the text prompt ‘A cartoon mouse landing on the moon with a view of planet Earth’ and the software will then do the rest.

Image-to-image in Stable Diffusion

Another feature found in Stable Diffusion is what is known as image-to-image generation. 

Here, the tool will use not only a text prompt but also an initial input image to create unique images based upon the input image. 

Put very simply, image-to-image is the process of taking a source image and altering it so that it matches the features and characteristics of a target image, or target images. You are essentially taking one image and transforming it into another one.