Transcending image generation boundaries

Let’s begin with a thought experiment. Imagine an art teacher telling your class of students a story about visiting a wonderful house with a big garden with old trees and beautiful flowers.

Now, the teacher gives you a piece of strange canvas with many dots (pixels of noise in an image). This mysterious piece of paper is a potential (latent) space of hidden forms you must find in your mental representation of the words (text) the teacher spoke. As you erase the dots and replace them with your ideas, you are dispersing them (diffusion). You obtain a small sketch of the objects you imagined. Your drawing is incomplete, and it’s a smaller view of what you thought. You just represented the main forms you saw. You downsampled your representation.

The fun now begins. You show each other your sketches. Although every drawing shows a house, not one is the same! Your teacher now provides incredible oil painting techniques to fill...