Training LoRAs on AI-Generated (Synthetic) Images: When It Works and When It Collapses

Every creator who expands one OC reference with Gemini (or mixes real photos with generated coverage) asks the same question: is training a LoRA on AI-generated images okay? The honest answer is neither always fine nor always cursed. Synthetic data can fill named gaps when curated and anchored; it collapses when you feed uncurated dumps, gens-of-gens, or conflicting identities into the trainer. This post owns that decision framework. The hands-on expansion workflow lives in creating a dataset from one image—here we cover when that workflow helps, when it fails, and what curation still owns.

What this guide covers (and what it does not)

Scope is the decision framework for training on synthetic images: when it helps, collapse modes, curation, hybrid real + synthetic, ethics, and an honest product placement. Out of scope: full Kohya hyperparameter recipes, anime tag taxonomy, and folder trees. You will leave with a when/when-not table, named failure modes, a copyable checklist, and a hybrid ablation recipe.

Related diagnosis: identity drift, overfitting vs undertraining, product geometry risks.

Quick answer

1. Use synthetic to cover named gaps real or consented images do not cover well.
2. Keep an anchor (original ref(s) or locked identity spec)—do not let generated volume outvote verified evidence.
3. Generate into a staging folder; reject defects; dedupe; then promote.
4. Compare at least anchor-only vs mixed under matched trainer settings before trusting a synthetic-heavy release.
5. There is no universal safe % of synthetic for image LoRAs—do not invent one.

When synthetic helps

One OC / one reference and no photoshoot

Fictional characters and illustrated designs often have no camera day. Community OC pipelines (for example Phosphene) describe generating roughly 20–50 AI refs as an initial model-specific trial range—label that as that guide’s range, not site law. The site one-image workflow already assumes expansion from a single strong ref into a diversity grid. That can work when gens stay on-spec and varied.

Privacy and consent boundaries

Synthetic expansion can reduce the need to scrape or share many photos of a real person, but it does not grant permission to invent a likeness of someone who did not consent. Creators who own one authorized selfie can add controlled variations without a second shoot—still keep the real photo as the identity anchor when likeness matters.

Consistency and coverage gaps

Fill named holes: missing three-quarter, side, full-body, outdoor lighting, alternate outfit. A 30-image real set missing two viewpoints may benefit more from four carefully verified additions than from 30 generated portraits. Style and pose LoRAs with no real corpus often must use synthetic—still curate for diversity.

When it fails (collapse modes)

Training on gens of gens (recursive loops)

Community pattern: train v1 on few refs → generate more → train v2 on those gens → repeat. That can improve if each round is heavily curated and still tied to originals; it collapses when each round feeds the next without a source of truth.

• Shumailov et al., Nature 2024 (AI models collapse when trained on recursively generated data): indiscriminate recursive training on model-generated data causes model collapse—tails of the original distribution disappear; defects compound. Demonstrated across LLMs, VAEs, and GMMs. They do not publish a safe % for character LoRA folders.
• Ahn et al., arXiv:2407.17493 (Model Collapse in the Self-Consuming Chain of Diffusion Finetuning): SD1.5 finetuned with LoRA on synthetic images from prior iterations; severe quality degradation; CFG scale affects collapse type/rate. Use for “diffusion + LoRA recursive loops degrade”—not as a folder-percentage law.

Uncurated dumps and artifact amplification

The LoRA learns the generator’s shortcomings too: hands, plastic skin, freckle noise, default faces, limited body diversity, composition habits. Defects that look minor on a contact sheet become repeated LoRA features. Prefer small clean synthetic sets over large artifact-heavy ones. Plastic skin or rigid pose often gets misread as “need more steps”—start with dataset defects; see overfitting vs undertraining.

Style collapse and the default pretty face

AI refs carry generator face bias; characters drift toward the same default pretty face if prompts are not varied. Photos beat gens when likeness to a real person matters. Style LoRAs with unvaried subjects or layouts learn one composition template, not a flexible style.

Identity drift and conflicting identities

Synthetic sets fail when accepted images disagree on eye shape, age, marks, or hairline, or when multiple lookalikes share one trigger. Mixing Midjourney dumps + Flux gens + random Civitai samples without an identity lock averages into mush. Primary guide: fixing identity drift.

Same-model leakage

If you generate training images with the same base you will later attach the LoRA to, the adapter may reinforce what the base already believes instead of teaching the target. Document generator ≠ training base when possible, or at least validate that synthetic adds independent coverage. Pick the training family with Flux vs SDXL.

Curation checklist (before anything enters the training folder)

1. Identity / object spec locked — must-keep traits written down.
2. Named gap — every candidate fills a coverage cell; reject pretty but redundant.
3. Technical integrity — no watermarks, heavy compression, borders, broken files.
4. Visual correctness — hands, eyes, teeth, limbs, text, reflections, seams; zoom in.
5. Target fidelity — reject changed bone structure, invented accessories, wrong materials, guessed tattoos/logos.
6. Dedup — exact and near-duplicate clusters vs synthetic and vs anchor.
7. Bias audit — one face, one body type, one camera angle dominating? Fix the matrix.
8. Caption honesty — describe final visible pixels; strip prompt leftovers. See the captioning guide.
9. Provenance — manifest (generator, seed if available, intended gap, accept/reject). Origin lives here, not as a mandatory caption token.
10. Staging → promote — never drop raw gens straight into the trainer path. Keep staging separate until approval; then pack with Kohya / Civitai packaging.
11. Rights / consent — see Ethics below.
12. Ablation gate — mixed must beat (or not regress vs) anchor-only on identity, control, diversity, artifacts.

# manifest.csv (example row)
filename,gap,generator,decision,reason
side_03.png,side_view_outdoor,Gemini,accept,fills missing outdoor side; identity held
pretty_12.png,none,Gemini,reject,near-dup of hero three-quarter

Hybrid real + synthetic (safe workflow)

1. 01

   Keep an anchor

   Real photos when likeness or product geometry matters. For pure OCs, keep the best first-pass refs or design sheets as non-negotiable identity evidence. Use separate folders: anchor/, synthetic_staging/, dataset/ (approved only).
2. 02

   Smallest useful synthetic subset

   No universal ratio. Control effective exposure (images × repeats × epochs) so synthetic does not silently dominate via higher repeats. Prefer gap-fill over volume.
3. 03

   Ablate, do not beauty-contest

   Train anchor-only vs mixed with matched base, rank, LR, schedule, and validation prompts/seeds. Score gap filled, identity held, prompt control, diversity across seeds, and artifact rate separately. Polished synthetic-only that loses identity = generator prior, not proven dataset quality.

Ethics and consent (real people)

How LoRA Dataset fits

Position: first-pass expander + captions → you still own curation, ablation, and ethics. Companion walkthrough: create a LoRA dataset from one image. Anime / Pony sets still need tag hygiene after expansion — see the WD14 / Danbooru guide.