Stable Diffusion Webgpu

Stable Diffusion WebGPU demo is a web-based application that allows users to generate images using the create-react-app framework. To access the application, users need to have JavaScript enabled and use the latest version of Chrome, which requires enabling the "Experimental WebAssembly" and "Experimental WebAssembly JavaScript Promise Integration (JSPI)" flags.

The application performs a series of inference steps to generate an image, with each step taking approximately 1 minute plus around 10 seconds for the VAE decoder to generate the image.

It is worth noting that having the DevTools open can slow down the process by approximately 2 times. The UNET model, which is responsible for image generation, only runs on the CPU due to better performance and accurate results compared to the GPU.For acceptable results, the minimum number of steps recommended is 20; however, for demonstration purposes, 3 steps would suffice.

The model files are cached, eliminating the need for repeated downloads.The application provides a user-friendly interface with options to load the model, run the image generation process, and view the result.

To address specific issues, an FAQ section is available, offering troubleshooting guidance.Despite running on a GPU, the webgpu implementation in onnxruntime is still in its early stage, causing some operations to be incomplete.

This leads to data being continuously transferred between the CPU and GPU, impacting performance. Multi-threading is not currently supported, and limitations in WebAssembly prevent the creation of 64-bit memory with SharedArrayBuffer.

The developer plans to address these issues through proposed spec changes and engine patches.The source code for Stable Diffusion WebGPU demo is available on GitHub, allowing users to run it locally.

Additionally, a patched version of onnxruntime is provided, enabling the use of large language models with transformers.js, although its reliability in all scenarios is not guaranteed.

The developer also plans to submit a pull request to the onnxruntime repository.