Object recognition beyond RGB

Abstract

Object recognition and the subproblem of land cover classification has been a key focus of computer vision research. An increasing number of devices have begun supporting the capture of images with additional bands beyond the standard RGB bands, including depth and other spectra such as near infrared. There is an opportunity to study the use of RGB images with depth and multispectral images to improve the accuracy of the object recognition and land cover classification. We do this by taking existing state-of-the-art object recognition models and modifying them to work with RGB images with depth. For land cover classification we present a novel model, LandNet, which allows varying the number of backbone feature extractors and the image bands in each. We also study the impact of adding the additional depth information, bands and the use of multiple feature extractors on the training and inference times of the models. We find that adding depth data did not show any benefits for object recognition but has little effect on the training and inference times. Utilizing multispectral images allows for improvements for the accuracy of land cover classification. Adding the additional bands in single feature extractor has no effect on the training and inference times, however using multiple feature extractors does increase the training and inference times. The results leads us to conclude that depth data has the potential to improve object recognition accuracy but a larger dataset than SUN RGB-D is required to demonstrate improved performance when using RGB and depth images. We can conclude that using multispectral images for land cover classification has tangible benefits