This summer we were fortunate to have two, undergraduate summer interns come in to help us out with different projects related to the clusters. The first intern to arrive was Haoda Wong from USC. Haoda had a good bit of experience with Linux systems so we had him help us do some experiments with some nodes we just bought to do 100Gb/s packet recording. The nodes each have two AMD Epyc processors, 1TB of RAM, 10x2TB of U.2 NVMe storage, and a 100Gb/s Mellabox VPI NIC. He did a nice write up of the work he did in SAND2019-10319.
The Epyc nodes had a few new features so the first thing we had Haoda do after setting up the hardware was run some benchmarks to get a better idea of how the system should be configured. He tried a few different OSs and hardware configs. One interesting observation was that the system was slightly faster when only half the memory sockets were filled (AMD docs had warnings about this). New Ubuntu kernels had slightly better performance in some benchmarks, but we were stuck with RHEL due to driver issues with Mellanox.
The U.2 storage performed very well. Haoda found that the drives were very fast and that we could get close to 20GB/s of streaming write performance by using Btrfs or XFS raids. Interestingly, ZFS didn't perform very well, possibly due to its complexity and the speed of the drives. Haoda also explored using SPDK to stream data to disk via kernel bypass, but given that we only needed to hit 12GB/s speeds for worst-case network capture, we decided to stick with plain i/o.
Haoda spent the rest of the summer fighting Mellanox drivers and tweaking settings to get the Ethernet NICs to run at 100Gbps speeds. After a great deal of searching he realized that while the userspace DPDK library could grab packets from the wire correcly, running the data through libpcap caused multiple memory copies in order to format the data for output. Writing the raw packets to disk and then reading them in a follower application made it possible to keep up with line rates. It was a frustrating journey, but I'm glad we figured it out.