Craig Ulmer

EMPRESS Metadata Harvesting

2017-11-01 faodel io hpc pub

Margaret Lawson and Jay Loffstead put a paper together for PDSW-DISCS about their work with EMPRESS. EMPRESS uses FAODEL to export data from a sim so that metadata harvesting services can extracts statistics for users. Margaret was the first person outside our regular group to use FAODEL (though it was called Faodail at this point), and had to face a lot of early reliability/usability issues.

  • Note: EMPRESS and EMPIRE are completely separate projects that don't have any involvement with each other.

Abstract

Significant challenges exist in the efficient retrieval of data from extreme-scale simulations. An important and evolving method of addressing these challenges is application-level metadata management. Historically, HDF5 and NetCDF have eased data retrievalby offering rudimentary attribute capabilities that provide basic metadata. ADIOS simplified data retrieval by utilizing metadata for each process' data. EMPRESS provides a simple example of the next step in this evolution by integrating per-process metadata with thestorage system itself, making it more broadly useful than single file or application formats. Additionally, it allows for more robust and customizable metadata.

Publications

  • PDSW-DISCS Paper Margaret Lawson, Jay Lofstead, Scott Levy, Patrick Widener, Craig Ulmer, Shyamali Mukherjee, Gary Templet, and Todd Kordenbrock, "EMPRESS: Extensible Metadata PRovider for Extreme-scale Scientific Simulations", Proceedings of the 2nd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems, November 2017.