ComfyUI

ComfyUI is an open source, node-based program that allows users to generate images from a series of text prompts. It uses free diffusion models such as Stable Diffusion as the base model for its image capabilities combined with other tools such as ControlNet and LCM Low-rank adaptation with each tool being represented by a node in the program.

History

''ComfyUI'' was released on GitHub in January 2023. According to comfyanonymous, the creator, a major goal of the project was to improve on existing software designs in terms of the user interface. The creator had been involved with Stability AI but by 3 June 2024 that involvement had ended and an organization called Comfy Org had been created along with the core developers. In July 2024, Nvidia announced support for ComfyUI within its RTX Remix modding software. In August 2024, support was added for the Flux diffusion model developed by Black Forest Labs, and Comfy Org joined the Open Model Initiative created by the Linux Foundation. As of November 2024, the project has 58.6k stars on GitHub. ComfyUI is one of the most popular user interfaces for Stable Diffusion, along with Automatic1111.

Features

''ComfyUI'''s main feature is that it is node based. Each node has a function such as "load a model" or "write a prompt". The nodes are connected to form a control-flow graph called a workflow. When a prompt is queued, a highlighted frame appears around the currently executing node, starting from "load checkpoint" and ending with the final image and its save location. Workflows commonly consist of tens of nodes, forming a complex directed acyclic graph. Node types include loading a model, specifying prompts, samplers, schedulers, VAE decoders, face restoration and upscaling models, LoRAs, embeddings, and ControlNets.

Several samplers are supported, such as Euler, Euler_a, dpmpp_2m_sde and dpmpp_3m_sde. Workflows can be saved to a file, allowing users to re-use node workflows and share them with other users. The file format for the workflows is in JSON and can be embedded in the generated images. Users have also created custom extensions to the base system which are exposed as new nodes, such as the extension for AnimateDiff, which aims to create videos. ''ComfyUI'' has been described as more complex compared to other diffusion UIs such as Automatic1111. A default node group is also included with the program.

As of December 2024, 1,674 nodes were supported. ''ComfyUI'' Supports multiple text-to-image models including, Stable Diffusion, Flux and Tencent's Hunyuan-DiT, as well as custom models from Civitai like ''Pony''.

LLMVision extension compromise

In June 2024, a hacker group called "Nullbulge" compromised an extension of ComfyUI to add malicious code to it. The compromised extension, called ComfyUI_LLMVISION, was used for integrating the interface with AI language models GPT-4 and Claude 3, and was hosted on GitHub. Nullbulge hosted a list of hundreds of ComfyUI users' login details across multiple services on its website, while users of the extension reported receiving numerous login notifications. vpnMentor conducted security research on the extension and claimed it could "steal crypto wallets, screenshot the user’s screen, expose device information and IP addresses, and steal files that contain certain keywords or extensions".

Nullbulge's website claims they targeted users who committed "one of our sins", which included AI-art generation, art theft, promoting cryptocurrency, and any other kind of theft from artists such as from Patreon. They claimed that they were "a collective of individuals who believe in the importance of protecting artists' rights and ensuring fair compensation for their work" and that they believed that "AI-generated artwork is detrimental to the creative industry and should be discouraged".

References

{{Reflist, refs=

{{cite web , author1=comfyanonymous , title=LICENSE , url=https://github.com/comfyanonymous/ComfyUI/blob/220afe33101a3874cf40e9cc221d27f8a8b1fae1/LICENSE , website=github , access-date=10 July 2024

{{cite web , author1=comfyanonymous , title=Initial commit. , url=https://github.com/comfyanonymous/ComfyUI/commit/220afe33101a3874cf40e9cc221d27f8a8b1fae1 , website=github , access-date=10 July 2024

{{cite web , author1=comfyanonymous , title=ComfyUI , url=https://github.com/comfyanonymous/ComfyUI , website=github , access-date=10 July 2024

{{cite web , last1=Harper , first1=Christopher , title=Nvidia's RTX Remix goes open source —chipmaker adds Rest API to interface with ComfyUI for AI remastering or generating new graphics in real time , url=https://www.tomshardware.com/pc-components/gpus/nvidia-rtx-remix-goes-open-source , website=Tom's Hardware , access-date=11 July 2024 , language=en , date=4 July 2024

{{cite web , author1=白鲸出海 , title=一家成都游戏公司，做出了一款千万月访问量的AI图像产品-36氪 , url=https://36kr.com/p/2787164625961859 , script-website=zh: 36氪 , language=zh , date=23 May 2024

{{cite web , author1=机器之心 , title=当韩国女团BLACKPINK进军二次元，清华叉院AI神器原来还能这么玩-36氪 , url=https://36kr.com/p/2520471518996233 , script-website=zh: 36氪 , language=zh , date=16 November 2023

{{cite web , author1=comfyanonymous , title=ComfyUI is now 4 months old! , url=https://blog.comfyui.ca/comfyui/update/2023/05/18/ComfyUi-is-4-months-old.html , website=ComfyUI blog , access-date=11 July 2024 , language=en , date=18 May 2023

{{cite web , title=ComfyUI 作者团队成立 Comfy Org- DoNews快讯 , url=https://www.donews.com/news/detail/8/4335267.html , script-website=zh: DoNews

{{cite web , last1=田口 , first1=和裕 , title=画像生成AI「Stable Diffusion」使い倒すならコレ！ 「ComfyUI」基本の使い方 (1/3) , url=https://ascii.jp/elem/000/004/184/4184300/ , website= ascii.jp , language=ja

{{cite web , last1=新 , first1=清士 , title=アニメの常識、画像生成AIが変える可能性「AnimateDiff」のすごい進化 , url=https://weekly.ascii.jp/elem/000/004/155/4155433/3/ , website= ascii.jp , language=ja

{{cite web , last1=田口 , first1=和裕 , title=Macで始める画像生成AI 「Stable Diffusion」ComfyUIの使い方 (3/5) , url=https://ascii.jp/elem/000/004/190/4190538/3/ , website= ascii.jp , language=ja , date= 27 March 2024

{{cite web , author1=しらいはかせ , title=画像生成AIを使い倒す！「Stability Matrix」で使えるWebUIを紹介【生成AIストリーム】 , url=https://forest.watch.impress.co.jp/docs/serial/aistream/1554982.html , website= Impress Watch , language=ja , date=18 December 2023

{{Cite web , last=Maiberg , first=Emanuel , date=2024-06-11 , title=Hackers Target AI Users With Malicious Stable Diffusion Tool on GitHub to Protest 'Art Theft' , url=https://www.404media.co/hackers-target-ai-users-with-malicious-stable-diffusion-tool-on-github/ , access-date=2024-06-14 , website=404 Media , language=en

{{cite book , last1=Phoenix , first1=James , last2=Taylor , first2=Mike , title=Prompt engineering for generative AI: future-proof inputs for reliable AI outputs at scale , date=2024 , publisher=O'Reilly , location=Beijing Boston , isbn=978-1098153434 , edition=First , section=AUTOMATIC1111 Web User Interface , quote = Advanced users may also want to explore ComfyUI, as it supports more advanced workflows and increased flexibility (including image-to-video), but we deemed this too complex for the majority of use cases, which can easily be handled by AUTOMATIC1111.

{{cite book , last1=Zhu , first1=Andrew , title=Using Stable Diffusion with Python: Leverage Python to control and automate high-quality AI image generation using Stable Diffusion , date=2024 , publisher=Packt Publishing , isbn=978-1835084311 , quote=ComfyUI is a node-based UI that utilizes Stable Diffusion. It allows users to construct tailored workflows, including image post-processing and conversions. It is a potent and adaptable graphical user interface for Stable Diffusion, characterized by its node-based design.

{{cite journal , last1=Guo , first1=Yuwei , last2=Yang , first2=Ceyuan , last3=Rao , first3=Anyi , last4=Liang , first4=Zhengyang , last5=Wang , first5=Yaohui , last6=Qiao , first6=Yu , last7=Agrawala , first7=Maneesh , last8=Lin , first8=Dahua , last9=Dai , first9=Bo , title=AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning , journal=International Conference on Learning Representations , date=May 2024 , arxiv=2307.04725 , url=https://iclr.cc/virtual/2024/poster/19044

{{cite web , author1=田口和裕 , title=画像生成AI「Stable Diffusion」の代替に? 話題の「FLUX.1」を試した (1/7) , url=https://ascii.jp/elem/000/004/214/4214157/ , website=ASCII.jp , language=ja , date=August 7, 2024

{{cite book , last1=Pérez-Colado , first1=Iván J. , last2=Freire-Morán , first2=Manuel , last3=Calvo-Morata , first3=Antonio , last4=Pérez-Colado , first4=Víctor M. , last5=Fernández-Manjón , first5=Baltasar , title=2024 IEEE Global Engineering Education Conference (EDUCON) , chapter=AI Asyet Another Tool in Undergraduate Student Projects: Preliminary Results , date=8 May 2024 , pages=1–7 , doi=10.1109/EDUCON60312.2024.10578883, isbn=979-8-3503-9402-3

{{cite web , author1=故渊 , title=7 年老显卡 GTX 1080 能跑，图片生成视频模型 Stable Video Diffusion 更新 - IT之家 , url=https://www.ithome.com/0/734/991.htm , website=ithome , date=November 25, 2023

{{cite arXiv , last1=Xue , first1=Xiangyuan , last2=Lu , first2=Zeyu , last3=Huang , first3=Di , last4=Ouyang , first4=Wanli , last5=Bai , first5=Lei , title=GenAgent: Build Collaborative AI Systems with Automated Workflow Generation -- Case Studies on ComfyUI , date=2 Sep 2024 , class=cs.CL , eprint=2409.01392

{{cite arXiv , last1=Gal , first1=Rinon , last2=Haviv , first2=Adi , last3=Alaluf , first3=Yuval , last4=Bermano , first4=Amit H. , last5=Cohen-Or , first5=Daniel , last6=Chechik , first6=Gal , title=ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation , date=2024 , class=cs.CL , eprint=2410.01731

{{cite web , last1=Wheatley , first1=Mike , title=Linux Foundation's latest initiative aims to promote 'irrevocable' open-source AI models , url=https://siliconangle.com/2024/08/12/linux-foundations-latest-initiative-aims-promote-irrevocable-open-source-ai-models/ , website=SiliconANGLE , date=12 August 2024

{{cite web , last1=Zeman , first1=Benjamin , title=Adobe Photoshop's Firefly vs. ComfyUI and Stable Diffusion , url=https://www.xda-developers.com/adobe-photoshops-firefly-vs-comfyui-and-stable-diffusion/ , website=XDA , language=en , date=6 December 2024

{{cite web , last1=Zeman , first1=Benjamin , title=How to build basic workflows in ComfyUI , url=https://www.xda-developers.com/comfyui-how-to-workflows/ , website=XDA , language=en , date=4 December 2024

{{cite book , last1=Hu , first1=Qihan , last2=Xu , first2=Zhenghui , last3=Du , first3=Peng , last4=Zeng , first4=Hao , last5=Ma , first5=Tongqing , last6=Zhao , first6=Youbing , last7=Xie , first7=Hao , last8=Zhang , first8=Peng , last9=Liu , first9=Shuting , last10=Zang , first10=Tongnian , last11=Wang , first11=Xuemei , chapter=CanFuUI: A Canvas-Centric Web User Interface for Iterative Image Generation with Diffusion Models and ControlNet , title=AI-generated Content , series=Communications in Computer and Information Science , date=2024 , volume=1946 , pages=128–138 , doi=10.1007/978-981-99-7587-7_11 , chapter-url=https://doi.org/10.1007/978-981-99-7587-7_11 , publisher=Springer Nature Singapore , isbn=978-981-99-7586-0 , language=en , quote=Currently, the most popular user interfaces for Stable Diffusion are Stable Diffusion WebUI and ComfyUI.

Open-source artificial intelligence
Free and open-source software
Free software programmed in Python