FluxInpaintPipeline returns an image with an irrelevant background

[levoz92]

Hi.

This is my code for the flux inpaint pipeline:

import torch
from PIL import Image
from diffusers import FluxInpaintPipeline
from diffusers.utils import load_image

pipe = FluxInpaintPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16)
pipe.to("cuda")
iter = 3

prompt = "A car parked by a beach, high quality, photorealistic"
img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"
source = load_image(Image.open('car-001.png'))
mask = load_image(Image.open('car_black_and_white.png'))
image = pipe(prompt=prompt, height=2048, width=1024, image=source, mask_image=mask, strength=0.8, num_inference_steps=40, guidance_scale=9.0, num_images_per_prompt=1).images[0]
image.save(f"flux_car_{iter}_inpainting.png")

The car's is originally parked in a parking lot. I am trying to inpaint it with a beach background but its returning the same car inpainted into similar scenarios as the original one.

This is the original:

These are the result:


s:

[asomoza]

I remember that at least the img2img had a weird behavior, like it needed a lot more strength to be comparable to the SDXL one, so probably don't expect a good inpainting without a properly trained model.

Also, at least with the SD models, to completely change the background you need a strength of 1.0 or 0.99.

Finally what're you're doing is outpointing and not inpainting, which is a lot more harder to achieve, you can read about it, but there's still people that have a really hard time doing outpaintings even with an "inpainting" trained model and good controlnets.

I haven't had the time to play with the inpainting flux pipeline, but I don't have really high hopes for it yet, but at least, your images have good quality.

[levoz92]

I didn't add controlnets to this. How can I go about doing that? I haven't seen any implementation of this pipeline with controlnets.

[asomoza]

there's a PR that it's still not merged, you can test it if you want.

[levoz92]

How do I apply it to the same logic as this?

zoe_detector = 'ZoeDetector'
controlnet_1 = 'destitech/controlnet-inpaint-dreamer-sdxl'
controlnet_2 = 'diffusers/controlnet-zoe-depth-sdxl-1.0'
controlnet_3 = 'TheMistoAI/MistoLine'
autoencoder_kl = 'madebyollin/sdxl-vae-fp16-fix'

pipe_1 = 'SG161222/RealVisXL_V4.0'
pipe_2 = 'OzzyGT/RealVisXL_V4.0_inpainting'

Lora_tag = 'LoRA'

lora_name_1 = f''
lora_name_2 = f''
lora_name_3 = f''
lora_name_4 = f''

refiner_name = ''

zoe = ZoeDetector.from_pretrained("lllyasviel/Annotators")

controlnets = [
    ControlNetModel.from_pretrained(controlnet_1, torch_dtype = torch.float16),
    ControlNetModel.from_pretrained(controlnet_2, torch_dtype = torch.float16),
]

vae = AutoencoderKL.from_pretrained(autoencoder_kl, torch_dtype=torch.float16).to(DEVICE)
pipeline_1 = StableDiffusionXLControlNetPipeline.from_pretrained(pipe_1, torch_dtype = torch.float16, variant="fp16", controlnet = controlnets, vae = vae).to(DEVICE)
pipeline_2 = StableDiffusionXLInpaintPipeline.from_pretrained(pipe_2, torch_dtype = torch.float16, variant = "fp16", vae = vae).to(DEVICE)

[asomoza]

that's kind of old and also Flux doesn't have the same grade of maturity in the controlnet ecosystem.

Probably you won't be able to reproduce the same effects with Flux, also because it's a distilled model, the params are really weird and sensitive, so even 0.1 can make the whole generation bad.

When I get to Flux I can see if it's doable, but I have a lot of stuff from SDXL and SD3 still to do/test and Flux takes like 10 times for me to work with (need to use the cloud), so probably until we have the whole quantization/loras/controlnets working, at least for me, I won't be able to test it like the other models and even with that, it will take 3 times more, this is probably also true for researchers, trainers and fine tuners.

[ukaprch]

Try reducing guidance scale:
The guidance scale controls how much the final image adheres to the text prompt. Consider the guidance scale as the overall prompt strength. The higher the guidance scale, the less leeway your model has to exercise creativity and may produce a lesser quality image.

[paulaserna16]

Hi, sorry, which version of diffusers do you have? I can't import the FluxInpaintPipeline from diffusers. I don't know why, it must be related to the diffusers version.

Thanks!

[asomoza]

Hi, you need to install from the source code until the next release.