Abstract:
Artificial intelligence jargon encompasses deep learning that learns by training a deep neural network. Optimization is an
iterative process of improving the overall performance of a deep neural network by lowering the loss or error in the network.
However, optimizing deep neural networks is a non-trivial and time-consuming task. Deep learning has been utilized in many
applications ranging from object detection, computer vision, and image classification to natural language processing. Hence,
carefully optimizing deep neural networks becomes an essential part of the application development. In the literature, many
optimization algorithms like stochastic gradient descent, adaptive moment estimation, adaptive gradients, root mean square
propagation etc. have been employed to optimize deep neural networks. However, optimal convergence and generalization on unseen
data is an issue for most of the conventional approaches. In this paper, we have proposed a variance adaptive optimization (VAdam)
technique based on Adaptive moment estimation (ADAM) optimizer to enhance convergence and generalization during deep
learning. We have utilized gradient variance as useful insight to adaptively change the learning rate resulting in improved
convergence time and generalization accuracy. The experimentation performed on various datasets demonstrates the effectiveness of
the proposed optimizer in terms of convergence and generalization compared to existing optimizers.
