A software architecture for a real-time big data system: a case study of a spectrum-sensing enabled whitespace database

Abstract

Due to the ever-growing need to process vast amounts of data in real-time, more and more tools which serve different needs in the real-time Big Data processing pipeline have sprung out. However, holistic industry accepted frameworks that address all real-time Big Data processing requirements across the entire pipeline have not yet been developed. More so, the area of dynamic spectrum access, has more and more devices connecting to previously unavailable radio frequency spectrum. This vastly growing number of devices need real-time orchestration on how they access this newly made available spectrum. The development of a real-time Big Data system in the realm of dynamic spectrum access as required by the Council for Scientific and Industrial Research served as a case study for this research.

This research provides a step in reaching an industry wide accepted software reference architecture which will be followed in the development of real-time Big Data systems. This is done through uncovering the most important quality/architectural requirements of realtime Big Data systems which such a reference architecture is to address. It is shown that all major software reference architectures (Java Enterprise Edition, AutoSar, Microsoft.Net, and others) were developed with emphasis placed on addressing a set of specific prioritised requirements. Hence this research uses this principle to propose a method to help in the development of software architectures and software reference architectures of real-time Big Data systems.

In this research, a case study is used to make inference on the general population of real time Big Data systems about the method proposed in this research. A mathematical ranking method is employed to prioritise software architecture requirements of a case study system and the results are compared with literature to increase the accuracy of the inference. Then architecture design and experiments were carried-out and presented to the Council for Scientific and Industrial Research as the client for acceptance, which would serve as validation. This was further validated by comparing the results of the case study to work done by other researchers. Having uncovered the most important quality attributes for realtime Big Data systems, the software architecture design process for such systems is simplified and fertile ground has been laid for the development of software reference architectures for real-time Big Data systems.