Now that the EMPIRE review is done, we've packaged up all our changes and made a new release of the software (DIO). We received permission from both Sandia and DOE to make these updates available to the rest of the work. You can see this branch at https://github.com/faodel/faodel. Here's the changelog:
Release Improvements
- Kelpie has rank-folding table (RFT) pool
- Kelpie has new experimental IOMs (IomHDF5, IomLevelDB)
- New MPISyncStart service for auto-filling node ids in configurations
- DirMan is now its own service (previously a child of opbox)
- Lunasa has templated containers for storing a bundle of items in an LDO. The Generic(Sequential|Random)DataBundle classes are optimized for cases where data is accessed (in order|out of order).
- Users can now define webhooks for rendering specific DataObject types
- Bootstrap Start/Stops are much more robust
- General build fixes for use with EMPIRE
Significant User Visible Changes:
- Common and services directories renamed to faodel-common and faodel-services
- Kelpie pools now have a "behavior" that controls how data is copied
- Some components/tools changed directories. Check your includes
Known Issues
- Kelpie behavior setting may not be fully implemented on remote operations
- Some tests hang when using OpenMPI with the openib BTL. Given that this BTL is deprecated, we recommend using OpenMPI with a different network component. FAODEL has been tested with the TCP BTL and the psm, psm2, and libfabric MTLs.
- FAODEL's libfabric transport is still experimental. It does not fully implement Atomics or Long Sends. While Kelpie does not require these operations, other OpBox-based applications may break without this support.
- On Cray machines with the Aries interconnect, FAODEL can be overwhelmed by a sustrained stream of sends larger than the MTU. To avoid this problem, the sender should limit itself to bursts of 32 long sends at a time.
- This version does not have support for ARM8 or POWER cpus.
Update
During the process of getting FAODEL integrated into Spack, there were some minor bug fixes pushed out in a 1.1811.2 release.