Release: https://github.com/mcmonkeyprojects/SwarmUI/releases/tag/0.9.5-Beta

Code: https://github.com/Alpha-VLLM/Lumina-Image-2.0

Models: https://huggingface.co/Alpha-VLLM/Lumina-Image-2.0

• Add img2img and Image Variations tabs to the Qt GUI by @monstruosoft

Release: https://github.com/rupeshs/fastsdcpu/releases/tag/v1.0.0-beta.120

Flex.1 began as a finetune of FLUX.1-schnell, which allows the model to retain the Apache 2.0 license. It is designed to be fine tunable with Day 1 LoRA training support in AI-Toolkit.

Abstract

Story visualization, the task of creating visual narratives from textual descriptions, has seen progress with text-to-image generation models. However, these models often lack effective control over character appearances and interactions, particularly in multi-character scenes. To address these limitations, we propose a new task: customized manga generation and introduce DiffSensei, an innovative framework specifically designed for generating manga with dynamic multi-character control. DiffSensei integrates a diffusion-based image generator with a multimodal large language model (MLLM) that acts as a text-compatible identity adapter. Our approach employs masked cross-attention to seamlessly incorporate character features, enabling precise layout control without direct pixel transfer. Additionally, the MLLM-based adapter adjusts character features to align with panel-specific text cues, allowing flexible adjustments in character expressions, poses, and actions. We also introduce MangaZero, a large-scale dataset tailored to this task, containing 43,264 manga pages and 427,147 annotated panels, supporting the visualization of varied character interactions and movements across sequential frames. Extensive experiments demonstrate that DiffSensei outperforms existing models, marking a significant advancement in manga generation by enabling text-adaptable character customization. The code, model, and dataset will be open-sourced to the community.

Paper: https://arxiv.org/abs/2412.07589

Code: https://github.com/jianzongwu/DiffSensei

Project Page: https://jianzongwu.github.io/projects/diffsensei/

Abstract

While diffusion models show extraordinary talents in text-to-image generation, they may still fail to generate highly aesthetic images. More specifically, there is still a gap between the generated images and the real-world aesthetic images in finer-grained dimensions including color, lighting, composition, etc. In this paper, we propose Cross-Attention Value Mixing Control (VMix) Adapter, a plug-and-play aesthetics adapter, to upgrade the quality of generated images while maintaining generality across visual concepts by (1) disentangling the input text prompt into the content description and aesthetic description by the initialization of aesthetic embedding, and (2) integrating aesthetic conditions into the denoising process through value-mixed cross-attention, with the network connected by zero-initialized linear layers. Our key insight is to enhance the aesthetic presentation of existing diffusion models by designing a superior condition control method, all while preserving the image-text alignment. Through our meticulous design, VMix is flexible enough to be applied to community models for better visual performance without retraining. To validate the effectiveness of our method, we conducted extensive experiments, showing that VMix outperforms other state-of-the-art methods and is compatible with other community modules (e.g., LoRA, ControlNet, and IPAdapter) for image generation.

Paper: https://arxiv.org/abs/2412.20800

Code: https://github.com/fenfenfenfan/VMix (Coming soon)

Project Page: https://vmix-diffusion.github.io/VMix/

Abstract

We present 1.58-bit FLUX, the first successful approach to quantizing the state-of-the-art text-to-image generation model, FLUX.1-dev, using 1.58-bit weights (i.e., values in {-1, 0, +1}) while maintaining comparable performance for generating 1024 x 1024 images. Notably, our quantization method operates without access to image data, relying solely on self-supervision from the FLUX.1-dev model. Additionally, we develop a custom kernel optimized for 1.58-bit operations, achieving a 7.7x reduction in model storage, a 5.1x reduction in inference memory, and improved inference latency. Extensive evaluations on the GenEval and T2I Compbench benchmarks demonstrate the effectiveness of 1.58-bit FLUX in maintaining generation quality while significantly enhancing computational efficiency.

Paper: https://arxiv.org/abs/2412.18653

Code: https://github.com/Chenglin-Yang/1.58bit.flux (coming soon)

Change Log for SD.Next

SD.Next Xmass edition: What's new?

While we have several new supported models, workflows and tools, this release is primarily about quality-of-life improvements:

• New memory management engine
  list of changes that went into this one is long: changes to GPU offloading, brand new LoRA loader, system memory management, on-the-fly quantization, improved gguf loader, etc.
  but main goal is enabling modern large models to run on standard consumer GPUs
  without performance hits typically associated with aggressive memory swapping and needs for constant manual tweaks
• New documentation website
  with full search and tons of new documentation
• New settings panel with simplified and streamlined configuration

We've also added support for several new models such as highly anticipated NVLabs Sana (see supported models for full list)
And several new SOTA video models: Lightricks LTX-Video, Hunyuan Video and Genmo Mochi.1 Preview

And a lot of Control and IPAdapter goodies

• for SDXL there is new ProMax, improved Union and Tiling models
• for FLUX.1 there are Flux Tools as well as official Canny and Depth models,
  a cool Redux model as well as XLabs IP-adapter
• for SD3.5 there are official Canny, Blur and Depth models in addition to existing 3rd party models
  as well as InstantX IP-adapter

Plus couple of new integrated workflows such as FreeScale and Style Aligned Image Generation

And it wouldn't be a Xmass edition without couple of custom themes: Snowflake and Elf-Green!
All-in-all, we're around ~180 commits worth of updates, check the changelog for full list

ReadMe | ChangeLog | Docs | WiKi | Discord

Release: https://github.com/invoke-ai/InvokeAI/releases/

Source: https://github.com/MangoLion/justaaa

Has anyone been using any great video models? I've used SVD and it's been... fine, but I see some really great videos online and don't know how they're generated. I'm thinking mostly Img2Txt style, in ComfyUI so I can generate an image and then animate it. (My perfect case would be something like multimodal, where I give it an image and a prompt, and a video would be spit out)

Abstract

We introduce OneDiffusion, a versatile, large-scale diffusion model that seamlessly supports bidirectional image synthesis and understanding across diverse tasks. It enables conditional generation from inputs such as text, depth, pose, layout, and semantic maps, while also handling tasks like image deblurring, upscaling, and reverse processes such as depth estimation and segmentation. Additionally, OneDiffusion allows for multi-view generation, camera pose estimation, and instant personalization using sequential image inputs. Our model takes a straightforward yet effective approach by treating all tasks as frame sequences with varying noise scales during training, allowing any frame to act as a conditioning image at inference time. Our unified training framework removes the need for specialized architectures, supports scalable multi-task training, and adapts smoothly to any resolution, enhancing both generalization and scalability. Experimental results demonstrate competitive performance across tasks in both generation and prediction such as text-to-image, multiview generation, ID preservation, depth estimation and camera pose estimation despite relatively small training dataset. Our code and checkpoint are freely available at this https URL

Paper: https://arxiv.org/abs/2411.16318

Code: https://github.com/lehduong/OneDiffusion?tab=readme-ov-file

Model: https://huggingface.co/lehduong/OneDiffusion

Project Page: https://lehduong.github.io/OneDiffusion-homepage/

Release: https://github.com/mcmonkeyprojects/SwarmUI/releases/tag/0.9.4.0-Beta