3. Electronic Theses and Dissertations (ETDs) - All submissions

Permanent URI for this communityhttps://wiredspace.wits.ac.za/handle/10539/45

Browse

Search Results

Now showing 1 - 1 of 1
  • Item
    Learning to backpropagate
    (2020) Eyono, Roy Henha
    The backpropagation algorithm is regarded as the de-facto standard for gradient optimization in artificial neural networks. Since the conception of the method, modifications of the algorithm have been proposed. Adaptive learning rate methods are examples of custom modification to the back propagation equation as they scale the gradient equation during training. Existing gradient modifications are largely based on theoretical fundamen tals of gradient optimization, but few methods optimize for modifications that are based on data. We are motivated by the idea of discovering better custom gradient update equations for gradient optimization. In this paper, we present our parametrized backpropagation learning frame work (PBLF) which learns modifications of the backpropagation gradient update equation for stochastic gradient optimization. We achieve this by optimizing parts of the backpropagation equation to produce custom gra dient update equations for gradient optimization. We evaluate our custom equations by training our target network on a validation dataset. In our dissertation, we provide empirical analysis and evidence to support PBLF as a competitive alternative to standard backpropagation. In our experiments, we report competitive empirical performances on CI FAR10 with our custom gradient update equations sampled from PBLF. Our data-driven method offers promising custom update equations for gradient optimization
Copyright Ownership Is Guided By The University's

Intellectual Property policy

Students submitting a Thesis or Dissertation must be aware of current copyright issues. Both for the protection of your original work as well as the protection of another's copyrighted work, you should follow all current copyright law.