FLUX LoRA Fine-Tuning: Border Outpainting and Pixel-Level Misalignment Issues in Image Editing · Issue #1685 · modelscope/DiffSynth-Studio · GitHub
Skip to content

FLUX LoRA Fine-Tuning: Border Outpainting and Pixel-Level Misalignment Issues in Image Editing #1685

Open

Description

@lllllialois

Why does the flux.2-klein-base-9b model, when fine-tuned via LoRA on pixel-aligned pre- and post-edited image pairs with prompts for an image editing task, exhibit slight border expansion (outpainting), a failure to preserve pixel-level structural consistency, and minor detail distortions during inference?

Activity

  1. tutao0123 commented on Sep 14, 2026

    @tutao0123
    Contributor

    This pattern often comes from train/infer mismatch on how the source–target pair is resized and conditioned, not from LoRA rank alone.

    Worth checking in order:

    1. Pair geometry — are pre/post images exactly the same H×W before encode? Letterbox vs center-crop vs stretch will teach the model a soft “border expand”.
    2. Encode path — if source and target go through VAE with different pad/crop, latent grids won’t share the same pixel lattice and you get the slight structural drift you describe.
    3. Noise / CFG — edit LoRAs are sensitive to train noise scale and to high CFG at inference; try identity or near-identity prompts with lower CFG and see if the border still grows.
    4. Dataset edge padding — constant-color pads around the canvas are a classic outpainting prior.

    A quick A/B: same-size center crop (no pad), train a short run, then infer with identical resolution + low CFG. If the halo shrinks, it’s preprocessing; if not, share your train YAML (resolution, bucket, noise, CFG) and one sample pair sizes and we can narrow further.

  2. lllllialois commented on Oct 14, 2026

    @lllllialois
    Author

    Thanks for the suggestions! The border expansion issue has been resolved after adjusting the preprocessing (ensuring consistent geometry). However, pixel-level alignment remains a challenge.

    I have verified that the training and inference settings are identical, but I’ve observed a significant discrepancy between frameworks using the official flux.2-klein-base-9b model:

    With the same prompt and identical parameters, DiffSynth results show much less deviation from the source image, whereas Diffusers produces significantly larger changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions