Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy

Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy

• 1.
  Rethinking Data Augmentation forImage Super-resolution : A Comprehensive Analysis and a New Strategy; CutBlur & MoA Jaejun Yoo*, Namhyuk Ahn*, and Kyung-Ah Sohn
• 2.
  Motivation * Various levelsof vision tasks High-level: Semantic recognition (e.g., classification, object detection) Mid-level: Super-pixel (e.g., segmentation) Low-level: Pixels, Edges, Colors (e.g., super-resolution, denoising) ????? MixUp, Cutout, CutMix …
• 3.
  Analysis on existingDA methods “Sharp transitions, mixed image contents or losing the relationships of pixels can degrade SR performance.” e.g., Cutout fails (discarding pixels) and every feature method fails (manipulation). Training curves when applied feature DA’s
• 4.
  Analysis on existingDA methods • DA methods in pixel space bring some improvements when applied very carefully.
• 5.
  Analysis on existingDA methods • DA methods in pixel space bring some improvements when applied very carefully. • Cutout: Original setting (drop 25% of of pixels in a rectangular shape) significantly degrades the performance because it erases spatial information too much. However, erasing tiny amount of pixels (0.1% random pixels) boosts the performance (2~3 pixels of 48x48 input patch) Cutout
• 6.
  Analysis on existingDA methods • DA methods in pixel space bring some improvements when applied very carefully. • Mixup & CutMix: Improvements of using CutMix are marginal. We suspect this happens because CutMix generates a drastic sharp transition between two different images. Improvements of using Mixup is better than CutMix but it still generates unrealistic image and affects to the image structure. Mixup CutMix
• 7.
  Analysis on existingDA methods • DA methods in pixel space bring some improvements when applied very carefully. • CutMixup: To verify our hypothesis, we combine benefits of Mixup and CutMix; CutMixup. CutMixup provides various boundary cases while minimizes the sharp transition by retaining partial cues as Mixup does. CutMixup
• 8.
  Analysis on existingDA methods • DA methods in pixel space bring some improvements when applied very carefully. • Blend & RGB permutation: To push further, we tried a constant blending and RGB channel permutation, which turn out to be very simple but effective strategies showing big performance enhancement (dB). Note that both methods do not incur any structure modification to an image. BlendRGB perm.
• 9.
  CutBlur
• 10.
  CutBlur • What doesthe model learn from CutBlur? • CutBlur prevents the SR model from over-sharpening an image and helps it to super-resolve only the necessary region. Super-resolution results of a model (EDSR) trained without CutBlur and its error residual (Δ) Error residual (Δ)Output
• 11.
  CutBlur • What doesthe model learn from CutBlur? • CutBlur prevents the SR model from over-sharpening an image and helps it to super-resolve only the necessary region. Super-resolution results of a model (EDSR) trained CutBlur and its error residual (Δ) Error residual (Δ)Output with
• 12.
  Improved generalization: over-sharpening •Super-resolution (SR) • Trained on ×4 scale factor dataset and tested on different scale factor (×2)
• 13.
  Improved generalization: over-smoothing •Denoising • Trained on severe noise (! = 70) & tested on mild noise (! = 30).
• 14.
  Improved generalization: over-removal •JPEG artifact removal • Trained on a mild compression rate & tested on a severe compression rate
• 15.
  Mixture of Augmentation(MoA) • During the training phase … • Randomly select single augmentation at every step. (among the curated DA list) • Apply it!
• 16.
  Comparison on diversebenchmark models and datasets • SRCNN (0.07M) – ECCV’14, CARN (1.14M) – ECCV’18, RCAN (15.6M) – ECCV’18, EDSR (43.1M) – CVPRW’17 • DIV2K (synthetic), RealSR (real-world) • Our method shows consistent improvement for different models (parameters) and datasets (different environments and size):
• 17.
  Code: https://github.com/clovaai/cutblur Paper: https://arxiv.org/abs/2004.00448 QRcode for code & paper For more details, please visit our website: