GLEAN: Generative LatEnt BANk

for Large-Factor Image Super-Resolution

Kelvin C.K. Chan 1      Xintao Wang 2      Xiangyu Xu 1       Jinwei Gu 3      Chen Change Loy 1
1 Nanyang Technological University, Singapore
2 Applied Research Center, Tencent PCG
3 SenseBrain

The GLEAN framework. In contrast to existing works that optimize only the latent vectors of the generator (i.e. StyleGAN), GLEAN proposes to use the generator as a dictionary and condition it on the convolutional features provided by the encoder. In this way, the generator receives additional guidance on the local structures, producing results with high fidelity and quality.

Highlights

  1. GLEAN adopts a novel idea of a Generative Latent Bank, leveraging a pre-trained GAN as a dictionary.
  2. By switching different GANs, GLEAN can be applied to various categories.

Materials



Paper


Codes

Abstract


We show that pre-trained Generative Adversarial Networks (GANs), e.g., StyleGAN, can be used as a latent bank to improve the restoration quality of large-factor image super-resolution (SR). While most existing SR approaches attempt to generate realistic textures through learning with adversarial loss, our method, Generative LatEnt bANk (GLEAN), goes beyond existing practices by directly leveraging rich and diverse priors encapsulated in a pre-trained GAN. But unlike prevalent GAN inversion methods that require expensive image-specific optimization at runtime, our approach only needs a single forward pass to generate the upscaled image. GLEAN can be easily incorporated in a simple encoder-bank-decoder architecture with multiresolution skip connections. Switching the bank allows the method to deal with images from diverse categories, e.g., cat, building, human face, and car. Images upscaled by GLEAN shows clear improvements in terms of fidelity and texture faithfulness in comparison to existing methods.

Results

1. Large Factor Super-Resolution


2. Image Retouching


Citation

@InProceedings{chan2021glean,
  author = {Chan, Kelvin CK and Wang, Xintao and Xu, Xiangyu and Gu, Jinwei and Loy, Chen Change},
  title = {GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution},
  booktitle = {Proceedings of the IEEE conference on computer vision and pattern recognition},
  year = {2021}

Contact

If you have any question, please contact Kelvin Chan at chan0899@e.ntu.edu.sg.