Download as PDF, PPTX
More Related Content
![[BMVC 2022] DA-CIL: Towards Domain Adaptive Class-Incremental 3D Object Detec...](https://cdn.slidesharecdn.com/ss_thumbnails/0916slides-221108165348-620eaec7-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Introduction to Diffusion Models on deep learning
Introduction to Diffusion Models on deep learning
- 1.
- 2.
- 3.
- 4.
- 5. 5 1 2 3 5 6 Introduction ➔ Reminder ongenerative model ➔ Diffusion Model vs VAE Denoising Diffusion Probabilistic Models ➔ Principle ➔ Forward and Reverse Diffusion ➔ Training and Sampling Example: Fashion MNIST ➔ Generation of Fashion MNIST DDPM improvements ➔ Beta scheduling and Variance learning ➔ Fast sampling ➔ Latent diffusion DDPM applications ➔ Text-to-image ➔ Other task : inpainting / outpainting / super-resolution
- 6. Rappel - VAE- 1 Source https://lilianweng.github.io/posts/2021-07-11-diffusion-models/ 6
- 7. 7 Rappel - VAE- 2 VAE COST FUNCTION Sampling process Training process metric Decoder - Gaussian - Mixture of gaussian
- 8. Rappel - GAN- 1 2 networks in opposition : - Generator - Discriminator Ideal solution : - Generator ～ P(x|z) - Discriminator = ½ Adding supervision concept in an unsupervised task ! 8 Data to train a classification network Source https://www.kdnuggets.com/2017/01/generative-adversari al-networks-hot-topic-machine-learning.html
- 9. 9 Rappel - GAN- 2 Sampling process Training process Generator Train discriminator Train generator GAN COST FUNCTION - Uniform
- 10. Rappel - GAN- 3 GAN convergence problems ! Vanishing gradient due to discriminant being too perfect, generator can’t train anymore Mode collapse due to generator learning only some good examples instead of the whole data distribution 10 True Data Generated Data Source https://lilianweng.github.io/posts/2017-08-20-gan/ No convergence due to the nature of the problem (MinMax) True data Generated data
- 11. 11 VAE vs GAN VAE GAN -Generate high quality data - Hard to train - Have more diversity How to compare generative models ?
- 12. VAE vs DPM 12 VAE Lowdimensional representation of the input. DPM Fixed encoder High dimensional representation of the input. Z Decoder
- 13. DPM - Landscape 13 Source: https://github.com/bentoml/stable-diffusion-bentoml Dhariwal & Nichol, 2021 Source : Dall-E 2
- 14. DDPM - Principle- 1 14 After the training the Diffusion Model will generate images from Gaussian noise:
- 15. DDPM - Principle- 2 15 There are three processes that characterize Diffusion Models: 1. Forward Diffusion Process 2. Reverse Diffusion Process 3. Sampling Process
- 16. DDPM - Principle- 3 16 Forward Diffusion Process This process will add noise to any image gradually 0 ≤ t ≤ T ; T is a hyperparameter
- 17. DDPM - Principle- 4 17 Forward Diffusion Process Examples of images at different times t Here we choose T=1000, but it can be different values (it’s an hyperparameter)
- 18. 18 Reverse Diffusion Process Wetrain a model to predict xt-1 from xt x0 is any image from the dataset DDPM - Principle - 5 a bit less noisy than xt a bit more noisy than xt-1
- 19. 19 Reverse Diffusion Process Thesame model must predict every xt-1 from xt DDPM - Principle - 6
- 20. 20 Sampling Process From arandom noise we can generate an image DDPM - Principle - 7
- 21. DDPM - ForwardDiffusion - 1 21
- 22. DDPM - ForwardDiffusion - 2 22
- 23. DDPM - ForwardDiffusion - 3 23
- 24. DDPM - ForwardDiffusion - 4 24
- 25. DDPM - ForwardDiffusion - 5 25
- 26. DDPM - ForwardDiffusion - 6 26 So we can sample a noised image at any time step directly from original image
- 27.
- 28. DDPM - ReverseDiffusion - 1 28 We predict only the mean, we know the rest.
- 29. DDPM - ReverseDiffusion - 2 29 We can predict xt-1 by predicting zt
- 30. DDPM - ReverseDiffusion - 3 30 A little bit of explanation (a tiny bit): https://lilianweng.github.io/posts/2021-07-11-diffusion-models/
- 31. DDPM - ReverseDiffusion - 4 31
- 32. DDPM - ReverseDiffusion - 5 32
- 33.
- 34. DDPM - Training- 0 34 https://arxiv.org/abs/2006.11239
- 35.
- 36. 36 DDPM - Training- 3 Uniform distribution Between 1 and T t = 50 x0
- 37. 37 DDPM - Training- 4 Gaussian distribution Same shape than x0 zt ( = ϵ ) x0 t = 50
- 38. 38 DDPM - Training- 5 x0 t = 50 zt
- 39. 39 DDPM - Training- 5 xt t = 50 zθ ( = ϵθ ) x0 t = 50 zt xt
- 40. 40 DDPM - Training- 6 zθ Loss x0 t = 50 zt zθ zt 2 xt
- 41. 41 DDPM - Training- 7 Loss Backward ∇θ Weight update
- 42. 42 DDPM - Training- 8 and repeat ! It was just 1 iteration.
- 43. 43 DDPM - Sampling- 0 https://arxiv.org/abs/2006.11239
- 44. 44 DDPM - Sampling- 1 Gaussian distribution Same shape than training dataset images xT
- 45. 45 DDPM - Sampling- 2 xT T zθ (xT ,T) ≈ zT Reminder:
- 46. 46 DDPM - Sampling- 3 Gaussian distribution Same shape than training dataset images z (noise)
- 47.
- 48. 48 DDPM - Sampling- 5 and repeat ! Don’t generate xT , replace it by xT-1 and T by T-1… and do it again T time.
- 49.
- 50. DDPM 50 DDPM vs VAEvs GAN VAE GAN - Generate high quality data - Hard to train - Have more diversity - Long sampling process
- 51. 51 DDPM improvements Improving thelog-likelihood metrics (Improved DDPM, 2021) Improving image synthesis Dhariwal & Nichol, 2021 Faster sampling (Denoising Diffusion Implicit Model, 2021 / Latent Diffusion Model, 2021)
- 52. Beta scheduling -Cosine scheduling (IDDPM, 2021) 52 0.02 0.0001 1000 Almost pure noise Strong noising
- 53. Variance learning (IDDPM,2021) 53 Limitations : “(...) learning reverse process variances (...) leads to unstable training and poorer sample quality compared to fixed variance.” (DDPM, 2020) 0.001 DDPM IDDPM the early stage of diffusion are very important
- 54. 54 Diffusion & reverseprocess (DDIM, 2021) Mr. Gaussian Mr. Data Limitations : “For example, it takes around 20 hours to sample 50k images of size 32 x 32 from a DDPM, but less than a minute to do so from a GAN on a Nvidia 2080 Ti GPU.” (DDIM, 2021)
- 55. 55 Extension of DDPM(DDIM, 2021) Generalisation to a bigger class of inverse process (non-Markovian) Important : Same network and training as a DDPM
- 56. 56 Generation process (DDIM,2021) DDPM DDIM can be computed !!!
- 57. 57 Finding a betterinverse process (DDIM, 2021) => DDIM FID
- 58.
- 59.
- 60. Latent diffusion :Concept of latent space (reminder) 60 Latent space of MNIST database for AE and VAE Source https://thilospinner.com/towards-an-interpretable-latent-space/ ● Similar objects are close to one another in the latent space ● Usually lower dimension than original data (therefore does compression as well) ● Usually impossible to visualize by a human
- 61. Latent diffusion :Concept of latent space (example) 61 Example : Word embedding Actor - Pierre Curie + Marie Curie ≈ Actress Turn sparse data (for instance words) into vectors
- 62.
- 63. Latent diffusion model 63 Latentspace Encoder Decoder Image space + n o i s e + d i f f u s i o n
- 64. Conditional diffusion A pictureof GENCI’s supercomputer Jean Zay. On the storage bays, a picture of the eponymous minister, with a background representing a simulation of a turbulent flow of liquid sodium, and a quote from Jean Zay’s memoirs. Alongside the bays, the cooling equipment with the logo of the manufacturer and the owner of the supercomputer. How to control the output of a diffusion model and make sure it generates what we want ? 64
- 65. Conditional diffusion :text → image A picture of GENCI’s supercomputer Jean Zay. On the storage bays, a picture of the eponymous minister, with a background representing a simulation of a turbulent flow of liquid sodium, and a quote from Jean Zay’s memoirs. Alongside the bays, the cooling equipment with the logo of the manufacturer and the owner of the supercomputer. Latent space 65
- 66. Conditional diffusion: text→ image 66
- 67. Conditional diffusion :cross-attention Stable Diffusion uses cross-attention to make the denoising process consistent with the provided sentence embedding 67 Source Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
- 68. Conditional diffusion :other method Spatial Self Attention Dense layer Standard Unet layer (conv, maxpool, upsample, …) 68
- 69. ➔ Inpainting ➔ Super-resolution ➔Outpainting Other tasks Diffusion models can solve a variety of tasks. We already know about image generation, as well as conditional image generation (for instance with a short paragraph describing the picture) Other tasks: 69 Source Lugmayr, Andreas, et al. "Repaint: Inpainting using denoising diffusion probabilistic models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
- 70. ➔ Thin mask ➔Right side mask for halving the image ➔ Every second row of pixels for alternating lines Inpainting through masking ➔ Wide mask ➔ Outer mask for expanding the image ➔ Every second pixel in both directions for super-resolution We can solve many of these tasks through the usage of a mask 70 Source Lugmayr, Andreas, et al. "Repaint: Inpainting using denoising diffusion probabilistic models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
- 71. Inpainting through masking 71 Stept : Source Lugmayr, Andreas, et al. "Repaint: Inpainting using denoising diffusion probabilistic models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
- 72. Inpainting through masking:step t 72 + n o i s e + d i f f u s i o n New artifacts added (in this coarse example, our diffusion model drew a sun), so we force the known background again! xt x0 xt-1
- 73. Inpainting through masking:step t 73 + n o i s e + d i f f u s i o n xt x0 xt-1 This operation does not take into account the generated information
- 74. Inpainting deharmonization 74 Picture deharmonization:the generated image has a satisfying texture but is wrong semantically. The suggested solution is to resample. Source Lugmayr, Andreas, et al. "Repaint: Inpainting using denoising diffusion probabilistic models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
- 75. Inpainting resampling 75 xt x0 noising diffusion ×mask × (1 - mask) + xt-1 resampling This loop is performed several times (a hyperparameter) before moving on the next step
- 76. Inpainting resampling 76 n isthe number of times the resampling loop was performed Disadvantage: the number of required denoising steps is much higher Advantage: it produces much more satisfying results Lugmayr, Andreas, et al. "Repaint: Inpainting using denoising diffusion probabilistic models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
- 77. 77 Sources Papers: - Deep UnsupervisedLearning using Nonequilibrium Thermodynamics (DPM) (https://arxiv.org/abs/1503.03585) - Denoising Diffusion Probabilistic Models (DDPM) (https://arxiv.org/abs/2006.11239) - Improved Denoising Diffusion Probabilistic Models (IDDPM) (https://arxiv.org/abs/2102.09672) - Denoising Diffusion Implicit Models (DDIM) (https://arxiv.org/abs/2010.02502) - Diffusion Models Beat GANs on Image Synthesis (https://arxiv.org/abs/2105.05233) - High-Resolution Image Synthesis with Latent Diffusion Models (LDM) (https://arxiv.org/abs/2112.10752) - Repaint: Inpainting using denoising diffusion probabilistic models (https://arxiv.org/pdf/2201.09865) - Diffusion Models in Vision: A Survey (https://arxiv.org/abs/2209.04747) - Diffusion Models: A Comprehensive Survey of Methods and Applications(https://arxiv.org/abs/2209.00796) Other ressources: - Lilian Weng’s article (https://lilianweng.github.io/posts/2021-07-11-diffusion-models) - Yang Song’s article (https://yang-song.net/blog/2021/score) - Outlier video (https://www.youtube.com/watch?v=HoKDTa5jHvg)
- 78. Question break #5& Practice 78
- 79. Épisode 15 : AI,droit, société et éthique ● Interprétabilité, reproductibilité, biais ● Cadre légal ● Privacy ● Session interactive Durée : 2h Next, on Fidle: Jeudi 23 mars, 14h00
- 80. To be continued... Nexton Fidle : Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) https://creativecommons.org/licenses/by-nc-nd/4.0/ Séquence 15 : AI, droit, société et éthique Jeudi 23 mars, https://fidle.cnrs.fr Contact@fidle.cnrs.fr https://fidle.cnrs.fr/youtube Merci !