Rapidly generating images through AI can greatly accelerate the design process for conceptual ideas. If designers can also have complete control over the composition, achieving the contour and perspective composition that they want to emphasize, that can have an accelerating effect on the entire design process.
The following introduces how to get Blender to generate initial concepts, allowing the AI to take over the rendering phase after execution. The AI tool used here is Stable Diffusion installed on the local machine, and operated through the ComfyUI interface. The entire workflow is based on this YouTuber – Mickmumpitz – As described.
Through his introduction, I learned that by exporting three patterns from Blender—depth map, contour map, and semantic segmentation map—one can use ControlNet in the Comfy UI as a reference for AI-generated images to generate the desired composition.
The operation is as follows:
First, let Blender output the depth map:
Next, a contour drawing is generated.
Then, by coloring the different blocks in an emission manner, the following colored block diagram can be obtained:
Next, go to the ComfyUI interface and enter the three images. Select the checkpoint you want to use, and add LoRA nodes to add more detailed control. You can find both the checkpoint and LoRA on the Civital website for download.
You can then use the AI Prompt to have the AI add details to such a composition, ultimately completing such an AI composite image.
Since it is generated by AI, by adjusting the Prompt and choosing from the SDXL style, different results can be obtained. By trying several times, you can find the generation image that best suits your needs.
Currently, AI is still prone to problems when generating text. This can be addressed through post-processing.
ComfyUI and this workflow can be found at Mickmumpitz Patreon website download: