ControlNet

Words struggle to pin down position

A prompt works well for deciding what gets drawn. But a request like "light coming from the left, the window frame crossing the upper-right third of the frame, the chair tilted slightly" gets mangled the instant it's turned into a sentence. Type the same sentence in ten times, and ten different layouts come out.

Fixing the seed does bring the same image back. But that pins down one specific image, not the composition itself. Change even a single word in the prompt, and the whole layout scatters again.

ControlNet doesn't try to solve this with words. It hands over the desired layout drawn as lines, directly. The prompt stays a prompt, and position gets decided by the lines instead.

What gets handed over is one layer of lines, not a picture

What gets handed over isn't a finished picture. It's an image that keeps only one layer of information — an outline drawing with just the edges pulled from a photo, a skeleton made of joint positions strung together, or a depth map that marks how far away each spot in the frame is using brightness.

There's a reason for keeping only one layer. Feed in a whole reference photo, and its color and material tag along too, wiping out any room for the model to generate something new. Keep only the edges, and shape holds while the slots for color and texture stay empty — and the prompt fills those slots in. It's the same as how ironing pins down one crease, not the whole cloth.

There's no need to draw the sketch by hand either. Separate tools exist that pull an outline or a skeleton out of a single reference photo, and their output can be used as-is. Of course, drawing the lines in yourself works too.

The original stays put, and only lines get added

ControlNet doesn't build a new image-generation model. It leaves the already-trained model completely untouched and locked, sets a copy of the same shape beside it, and trains only that copy. The copy learns to read the sketch and turn it into a signal that says "there's a line right here."

The signal the copy produces gets added in at every step where the original model refines the image. At first this signal is held down so it has no effect at all, and as training goes on, it gets opened up gradually to start pulling its own weight. It's the same as adding just the fold, without ever touching the weave of the fabric.

This structure means a separate side branch can be built for each kind of sketch and swapped in whenever it's needed.

How hard, and how far, to grip

How closely the sketch gets followed is set by a single dial. Grip it hard, and the image doesn't stray an inch from the lines, but it turns stiff and whatever's outside the lines comes out thin. Loosen the grip, and the image turns natural again, at the cost of the composition drifting a little. It's the same difference as pressing the iron down long versus barely grazing it.

When to grip and when to let go can also be set. An image starts as a blurry mass and sharpens over several stages, with the broad layout settled in the early stages and fine detail settled in the later ones. So gripping only the early stages and releasing the later ones keeps the composition while letting the fine detail come out freely.

Choosing what to pin down

Pull different kinds of sketches from the same photo, and the results diverge sharply. Use the outline, and the original's silhouette survives almost intact; use only the skeleton, and just the pose matches while everything else is generated completely fresh. Use only the depth map, and only the spatial layout survives, with the prompt deciding what actually occupies it.

So the order that works is deciding what to keep and what to change first, then picking the kind of sketch to match. Pin down too much, and there's no room left to generate anything new; pin down too little, and there was no point pinning anything down at all.