- Identity drift
- Troubleshooting
- Character LoRA
- IP-Adapter
Identity Drift in Character LoRAs: Causes and How to Fix It
Why a character LoRA gives a different face on every seed, how to diagnose it with a four-framing grid test, and the dataset and training fixes that work.
You trained a character LoRA, typed the trigger word, and got someone who looks like your character’s cousin. The eyes are close. The hair is right. But it is not them, and the face changes again on the next seed. That is identity drift, and it is the single most common failure in character LoRA training. This guide explains where drift comes from, how to diagnose which cause you are hitting, and the fixes that actually work, from dataset repair to training settings.
What is identity drift?
Identity drift is when a trained model produces a subject whose facial features, proportions, or apparent age vary between generations or diverge from the reference. Unlike a LoRA that simply failed to learn, a drifting LoRA produces plausible faces that are inconsistently almost right. It is a data problem far more often than a training problem.
The six causes of identity drift
| Cause | How it looks in outputs | Where it comes from |
|---|---|---|
| Inconsistent source images | Features blend into an average face | Photos from different ages, angles, filters, or even different people |
| Too few angles | Face collapses when the prompt asks for a profile or high angle | A dataset of mostly front-facing shots |
| Face-swap or IP-Adapter artifacts | Waxy skin, mismatched lighting on the face, blurred hairline | Datasets built by pasting a face onto other bodies |
| Captions describe the face | Trigger word alone gives a generic face | Auto-captioners that list eye color, jawline, freckles |
| Under- or over-training | Either weak resemblance or a rigid, over-sharpened copy of one photo | Wrong step count or learning rate for the dataset size |
| Rank too low for the model | Fine details like eye shape never settle | Network dim of 4 or 8 on a detailed face |
How to diagnose which cause you have
Run this grid test before changing anything. Fix the seed, keep the LoRA weight at 1.0, and generate the same prompt across four framings: close-up front, three-quarter medium shot, profile, and full body.
- Face is stable at close-up but falls apart at profile or full body → too few angles and framings in the dataset.
- Face is different on every seed even at close-up → inconsistent sources or captions that describe the face.
- Face is right but skin looks plastic and lighting on the face does not match the scene → face-swap artifacts in the source images.
- Every output reproduces one training photo’s pose and background → over-training or too few images.
- Resemblance is weak at every framing → under-training or rank too low.
Fixes, in order of impact
- 01
Audit the dataset against one reference
Pick the single best image as ground truth. Put every other image next to it and reject any where eye spacing, nose bridge, jawline, or apparent age differ. Be ruthless. A drifting LoRA usually traces back to three or four inconsistent images out of twenty. - 02
Replace face-swapped images with generated variations
Face-swap and IP-Adapter pipelines paste a face onto a body without re-rendering the lighting or skin, and the trainer learns those seams. Regenerate those images from the reference with an identity-preserving generator instead. Our guide to building a dataset from one image covers the workflow. - 03
Fill the angle and framing gaps
Count your images by angle and framing. If fewer than a quarter are three-quarter or profile views, or fewer than three are full body, add them. Aim for the diversity grid in how many images a LoRA needs. - 04
Rewrite captions to remove permanent traits
Strip eye color, hair color, face shape, and skin tone from every caption. Keep pose, wardrobe, lighting, framing, and setting. Make sure the trigger word is first. See the captioning guide for the full rule. - 05
Retrain with checkpoint sampling
Save a checkpoint every epoch and generate the same four-framing grid at each. Pick the earliest epoch where all four framings hold. If none do, raise rank from 16 to 32 or extend training by 20 percent, not both at once.
LoRA vs IP-Adapter vs InstantID for consistent characters
Trainers sometimes reach for reference-image adapters instead of training a LoRA, or use them to build the dataset. Each has a role, and understanding them explains why adapter-built datasets drift.
| Method | How it works | Consistency | Best for |
|---|---|---|---|
| Trained LoRA | Low-rank weights learned from 15–30 images | High across angles and styles once trained on a clean set | Recurring characters, commercial projects, style flexibility |
| IP-Adapter | Injects CLIP image features at inference; about 22M parameters | Moderate; drifts on new angles and struggles with fine identity | Quick one-off style or composition transfer |
| InstantID | Zero-shot face identity conditioning with a face encoder | Good for frontal faces; weaker on profiles and bodies | Fast portraits without training |
| LoRA trained on generated dataset | LoRA trained on identity-consistent variations of one reference | High, and controllable through captions | Original characters with only one reference image |
Why adapters drift
Training settings that reduce drift
| Setting | Drift-prone value | Safer value | Why |
|---|---|---|---|
| Network rank (dim) | 4–8 | 16–32 | Faces need capacity for fine detail |
| Learning rate | 5e-4 or higher | 1e-4 (UNet), 5e-5 (text encoder) | High rates overshoot and memorize |
| Total steps | Under 800 or over 4,000 | 1,500–3,000 (SDXL), 1,000–2,500 (Flux) | Both extremes hurt likeness |
| Caption dropout | 0 | 0.05–0.1 | Strengthens the trigger word binding |
| Resolution bucketing | Off, mixed sizes | On, 1024 base | Prevents distortion from forced crops |
Preventing drift from the start
The cheapest fix is a dataset that never drifts. That means one identity specification applied to every image, no face-swapped sources, planned angle coverage, and captions that follow the caption-what-changes rule. LoRA Dataset was built around this problem: it extracts a structured identity profile from one reference and enforces it while generating 20+ variations, then captions each image with the trigger token first and permanent traits omitted. The result is a dataset where every image agrees with the reference before you spend a single training step.
Frequently asked questions
Why does my LoRA look like my character but not exactly?
The model is averaging inconsistent features across your training images. Audit every image against one reference and remove any where eye spacing, nose, jawline, or apparent age differ, then retrain.
Does increasing LoRA weight above 1.0 fix identity drift?
Rarely. Weights above 1.0 amplify whatever the LoRA learned, including the inconsistency, and usually add over-sharpening artifacts. Fix the dataset instead.
Is IP-Adapter a good way to build a LoRA dataset?
It is risky. IP-Adapter re-interprets identity on every generation and pastes face features without re-rendering lighting, so images disagree with each other and carry seams. An identity-preserving generator that enforces a fixed trait profile produces more consistent sets.
What network rank should I use for a character LoRA?
Start at 16 for Flux and 32 for SDXL or SD1.5. Ranks of 4 to 8 often lack capacity for fine facial detail, which shows up as features that never settle.
How do I test a LoRA for identity drift?
Fix the seed and generate the same prompt at four framings: close-up front, three-quarter medium shot, profile, and full body. Compare all four against your reference. Drift that appears only at profile or full body means the dataset lacks those angles.
Sources and further reading
- Ye et al., "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models" (arXiv, 2023)
- Wang et al., "InstantID: Zero-shot Identity-Preserving Generation in Seconds" (arXiv, 2024)
- Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (arXiv, 2021)
- kohya-ss/sd-scripts: network_dim, learning rate, and caption dropout options