Comparison of Hamiltonian Monte Carlo Variants in Bayesian Deep Learning

Abstract

This research investigates the integration of Bayesian inference (BI) with deep learning (DL) models, focusing on Bayesian Deep Learning (BDL) as a solution to overfitting and uncertainty in predictions. By employing posterior probability distributions for uncertainty quantification, BDL enhances decision-making and reliability. Despite various posterior density estimation techniques, including Hamiltonian Monte Carlo (HMC) and its variants, a comprehensive comparative analysis of their performance, computational efficiency, and scalability in BDL remains lacking. This study compares three Bayesian Neural Network (BNN) models using different HMC variants—standard HMC, No-U-Turn Sampler (NUTS), and Stochastic Gradient HMC(SGHMC)—for regression tasks on the California Housing Prices dataset. Metrics such as mean squared error (MSE), root mean squared error (RMSE), and mean absolute error (MAE) were used to assess predictive accuracy, alongside training times to measure computational efficiency and scalability across varying dataset sizes. Results indicate that BNNSGHMC outperforms the others in predictive accuracy, computational efficiency, and scalability, particularly with larger datasets. This research provides recommendations for practitioners, emphasising the use of BNN-SGHMC for larger datasets, and contributes valuable insights to the field of BDL, paving the way for future studies on advanced HMC techniques.