Processing Particle Data Flows with SmartNICs

2022-09-23 Fri

pub smartnics hpc

Our ASCR SmartNIC project published the following unclassified unlimited release (UUR) paper.

Abstract

Many distributed applications implement complex data flows and need a flexible mechanism for routing data between producers and consumers. Recent advances in programmable network interface cards, or SmartNICs, represent an opportunity to offload data-flow tasks into the network fabric, thereby freeing the hosts to perform other work. System architects in this space face multiple questions about the best way to leverage SmartNICs as processing elements in data flows. In this paper, we advocate the use of Apache Arrow as a foundation for implementing data- flow tasks on SmartNICs. We report on our experiences adapting a partitioning algorithm for particle data to Apache Arrow and measure the on-card processing performance for the BlueField-2 SmartNIC. Our experiments confirm that the BlueField-2's (de)compression hardware can have a significant impact on in- transit workflows where data must be unpacked, processed, and repacked.

Publication

HPEC Paper Jianshen Liu, Carlos Maltzahn, Matthew L. Curry, and Craig Ulmer, "Processing Particle Data Flows with SmartNICs" in IEEE High Performance Extreme Computing, September 2022.
arXiv:2210.06827 The extended version of this paper on arXiv. DOI: https://arxiv.org/abs/2210.06827