EMPRESS Metadata Harvesting

We published an unclassified unlimited release (UUR) paper.

Abstract

Significant challenges exist in the efficient retrieval of data from extreme-scale simulations. An important and evolving method of addressing these challenges is application-level metadata management. Historically, HDF5 and NetCDF have eased data retrievalby offering rudimentary attribute capabilities that provide basic metadata. ADIOS simplified data retrieval by utilizing metadata for each process' data. EMPRESS provides a simple example of the next step in this evolution by integrating per-process metadata with thestorage system itself, making it more broadly useful than single file or application formats. Additionally, it allows for more robust and customizable metadata.

Publications

• PDSW-DISCS Paper Margaret Lawson, Jay Lofstead, Scott Levy, Patrick Widener, Craig Ulmer, Shyamali Mukherjee, Gary Templet, and Todd Kordenbrock, "EMPRESS: Extensible Metadata PRovider for Extreme-scale Scientific Simulations", Proceedings of the 2nd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems, November 2017.