INTRODUCTION

Deep neural network artificial intelligence (AI) has brought powerful pattern recognition capabilities to various applications in a broad span of industries. Setting up complex image interpretation and recognition software no longer requires deep expertise in machine vision feature selection. Instead, the technical challenge has been greatly simplified to the process of acquiring plenty of high-quality labeled image data and applying supervised gradient descent learning on popular architectures like the deep convolutional neural network using open source software frameworks like Tensorflow and Pytorch. While it is now easy to set up a deep neural network classifier within a few hours by following one of many tutorial instructions online, it remains challenging to ensure that the deep neural network is robustly well-trained for all kinds of data. If not well-configured, gradient learning often yields suboptimal classification and sometimes just fails to converge. This course focuses on the best practices in designing and configuring gradient learning for deep neural networks. We first introduce the methodology of gradient learning and backpropagation and highlight where gradient learning commonly fails. We review common training loss functions and regularization strategies which improve the convergence of gradient learning. With a good understanding of these fundamentals, we will study the motivation and implementation of input, weight and activation normalizations and clipping techniques that have been commonly used to stabilize gradient learning across multiple different network architectures. We will discuss a numerical technique to check gradients to assess the success of gradient learning. Finally, we will study methods to enhance learning convergence through adaptive learning algorithms.