Final project for ELEC 825 Machine Learning/ Deep Learning. A research project into using cGANs to remove snow distortion from input images.
In this paper, we propose an alternative method for single image snow removal using a more general and less task specific architecture than most state of the art methods. The proposed architecture aims to learn a general transition map between two images and is commonly used for tasks like transforming an aerial view image to a map. Current state of the art architectures involve multiple layers of snow identification and other network structures specific to the snow removal task. Using a general transition map architecture allows for easier implementation and a wider field of applications. A general adversarial network (GAN) and more specifically a pix2pix GAN architecture trained on the Snow100K dataset was implemented to achieve this goal. This solution showed promising results with a peak signal-to�noise ratio (PSNR) of 23.2839 and a structural similarity index (SSIM) 0.7697 and warrants additional investigation to see the limit in its applicability to the task of removing snowfall in images.
The contribution of this repository is to continue exploring methods of performing the task of weather removal. Most other generator structures such as DesnowGAN, have components designed for snow removal, followed by residual generation to fill the image. The pix2pix GAN is designed for general image translations and learns the translation necessary based on the provided images. The results show that simply learning the translation provides decent performance and further research into learning complex image translations could be combined with existing methods to further improve results.
Main analysis and training was performed using the Snow100K dataset. The dataset can be accessed here. Includes 50K training and 50K testing images of synthetically created snowy images and their groud truth images.
a) synthetic input b) generated snow removal c) ground truth
Main file used to train the cGAN, runs in the standard python console. Uses the .npz files generated by "npz_data_generator.py". Contains an experiemntal section to change many charcateristics of how the layers of the GAN are organized, including batch size, epoch count, downsampling method, batch normalization use, and dropout. This script will output network weight checkpoint files and example generated images on regular intervals.
File used to generate .npz files for input data into the cGAN. The synthetic and ground truth images are both taken from their stored directories and loaded into a numpy .npz file for quicker and easier loading into the cGAN. This way image files can be loaded once and accesses quicker.
Jupyter notebook that will calculate psnr and ssim results for the network weight files generated by "main_GAN_trainer". Will generate its own test images from a given file path containing images never seen before by the cGAN. Outputs the results to .xlsx files for easier analysis using excel.
A folder containing various scripts that were used to pre-process or create data to be used by the cGAN. Each file contains an outline at the top of the file and is well commented.
Riley Cooper - Electrical and Computer Engineering
Jason Harris - Electrical and Computer Engineering
Liam Horton - Mechanical and Materials Engineering
Francesco Marrato - Electrical and Computer Engineering
Full report is included which cover ours groups research, tests, and analysis.