Run on <4gb Vram
#6
by Hackerdada - opened
With a slower SSD, which stage becomes the main bottleneck?
With a slower SSD, Compute time will not be the bottle neck. The block transfer will be the main bottleneck, which you have to face during all the text encoder, transformer and vae model.
Roughly how many times longer does the generation time become for the same task?
During data transfer, is the GPU computing, or is it waiting/idle?
Why not keep it in memory instead of putting it on SSD?
- Depends on gpu computation time and block transfer time. If block transfer time < gpu computation time, then GPU does not wait. It will be exact same as if you are running the model in larger gpu.
- During data transfer, the GPU computes previous blocks. If 2 block can fit into your gpu, one block is being computed, then evicted, next block starts computing, and on the same time one block is getting transferred from ram to vram. But if block transfer time < gpu computation, then GPU will wait.
- Already blocks are being fetched from disk to ram. So I put the blocks on RAM before putting them on GPU. But as my RAM is not also enough to hold the model, I have to do the streaming from disk to ram to GPU. Or Disk to GPU directly. If the code finds out that you have enough RAM, then all the blocks are permanently kept on RAM. This is exactly same as enable_model_cpu_offload() function in diffusers in that case.