Archiving Visibility Data Using Lossy Baseline-Dependent SVD Techniques

Abstract

Modern radio interferometer arrays, such as the MeerKAT [1], the Australian Square Kilometre Array Pathfinder (ASKAP) [2, 3], the Low Frequency Array (LOFAR) [4], the Murchison Widefield Array (MWA) [5, 6], and the upcoming Square Kilometre Array Observatory (SKAO) [7], generate large volumes of data due to their high temporal and spectral resolutions, large number of baseline configurations, and wide bandwidths. Managing these data volumes poses substantial challenges in terms of storage and processing. To address the growing costs, averaging techniques are widely used to reduce data sizes. However, averaging leads to signal loss in radio interferometric images, resulting in smeared or blurred source emissions and reduced source amplitudes. Moreover, the extent of this smearing is baseline-dependent, as the signal phase depends on baseline length. Specifically, longer baselines are more affected than shorter ones. This is addressed by Baseline Dependent Averaging (BDA), which applies variable averaging intervals - longer for shorter baselines and shorter for longer baselines. BDA achieves high data volume reduction since radio interferometers generally have more shorter baselines, which can be aggressively averaged with minimal smearing effects. However, BDA changes the time-frequency grid structure of the data, making it incompatible with the standard storage format in the field, the Measurement Set (MS). A promising approach to data compression was presented by Atemkeng et al. [8], who developed a compression technique based on Singular Value Decomposition (SVD). This approach exploits the inherent structure of raw visibility data, representing it as a low-rank matrix approximation where each component corresponds to a specific Fourier component of the sky distribution. By approximating the data with a reduced rank, the essential features of the original data can be captured using fewer components, effectively reducing data size. In this work, we build on the methods introduced by Atemkeng et al. [8], specifically evaluating the effectiveness of SVD in compressing large volumes of data while preserving image quality and data fidelity for long-term archival. Although our study focuses on the MeerKAT telescope, the approach can be adapted for use with any other radio telescope. Our findings demonstrate that for a bright point source (1 Jy), whether located at the phase centre or away from it, the data features can effectively be captured using a single component, recovering over 99.90% of the source amplitude and achieving a data size reduction of over 97%. For fields with multiple sources, the features can be fully captured using 3-4 components out of 24, recovering over 99.90% of the source amplitude for a source at the edge of the Field of View (FoV), which is around 1.1 deg for the MeerKAT at a frequency of 1.4 GHz. This results in a data size reduction of over 91%. Additionally, we found that the source or field direction does not impact SVD compression. On the other hand, the Signal to Noise Ratio (SNR) significantly affects SVD compression. For sources with low SNR or faint sources, all components are required to recover more than 97% of the source amplitude, making the compression ineffective. In such scenarios, it would be more advantageous to first denoise the data or to use BDA.