How to fix identity drift in a character LoRA
Identity drift is when a character LoRA produces faces that look related but not the same from one generation to the next. It almost always starts in the dataset: inconsistent source identity, too few angles, or captions that describe fixed traits. Fix the dataset first, then adjust rank, learning rate, and steps if drift remains.
Published 5 September 2026 · 3 min read
What does identity drift look like?
- Eye spacing, nose width, or jawline change between generations while the overall look stays.
- The face is convincing head-on but becomes a different person at three-quarter or profile angles.
- Apparent age shifts by several years depending on the prompt.
- Hair colour or skin tone wanders toward the base model average.
- The face changes when you change the outfit or background, which means those were entangled with identity.
Which dataset problems cause drift?
Lay the dataset out as a contact sheet and look at it as a stranger would. If you cannot immediately say these are all the same person, the model cannot either. Common causes:
- Source images are themselves AI-generated with different seeds and no shared anchor, so each one is a slightly different person.
- All images are frontal, so the model has no information about the face in depth.
- Lighting varies so much that skin tone appears different across the set.
- Captions name eye colour, face shape, or other fixed traits, so those attach to words instead of the trigger.
- One or two outliers slipped in: a different person, a heavy filter, an extreme expression.
How do you fix the dataset?
Anchor every image to a single verified reference. Generating variations from one photo, rather than prompting fresh images for each pose, keeps the identity constant by construction. It is why LoRA Dataset starts from exactly one image.
Cover the head in depth: at least a few three-quarter views on each side and one profile. Keep lighting varied but not extreme. Remove outliers even if that leaves you with fewer images; the image-count guide explains why smaller and consistent beats larger and mixed. Finally, strip fixed identity traits from the captions.
Which training settings help, and when?
Only after the dataset is clean should you touch training settings. Match the symptom to the change:
| Symptom | Adjustment |
|---|---|
| Right from the front, wrong at an angle | Add angled images. No training setting substitutes for missing data. |
| Face looks averaged or soft | Lower repeats or epochs; you may be overtraining on an inconsistent set. |
| Trigger word barely changes the output | Raise total steps modestly, or check the trigger spelling in every caption. |
| Base model forgets how to draw other people | Add regularisation or class images, the prior-preservation idea from DreamBooth. |
| Fine facial detail missing at every angle | Raise the LoRA rank one step. Rank controls how much detail the adapter can store. |
How do you test for drift?
Pick four or five prompts that cover angles, lighting, and framing, fix the seeds, and render the same grid for every saved epoch. Kohya sd-scripts can save a checkpoint each epoch, so you can see exactly where identity locks in and where it starts to smear. Choose the epoch just before quality declines rather than the last one.
Frequently asked questions
- Is identity drift a LoRA problem or a dataset problem?
- Usually the dataset. A LoRA can only learn what is consistent across its images; if the identity is not consistent, the adapter learns an average. Fix the data before tuning rank or learning rate.
- Can I fix drift by raising the LoRA weight at generation time?
- Pushing the weight above 1.0 exaggerates whatever the LoRA learned, including the averaged face and any entangled backgrounds. It masks drift for one prompt and makes it worse for others.
- How do I know when the dataset is consistent enough?
- Show the contact sheet to someone who has never seen the character. If they can pick out the same person in every frame without hesitation, the dataset is ready.