Margaret Lawson and Jay Loffstead put a paper together for PDSW-DISCS about their work with EMPRESS. EMPRESS uses FAODEL to export data from a sim so that metadata harvesting services can extracts statistics for users. Margaret was the first person outside our regular group to use FAODEL (though it was called Faodail at this point), and had to face a lot of early reliability/usability issues.
Significant challenges exist in the efficient retrieval of data from extreme-scale simulations. An important and evolving method of addressing these challenges is application-level metadata management. Historically, HDF5 and NetCDF have eased data retrievalby offering rudimentary attribute capabilities that provide basic metadata. ADIOS simplified data retrieval by utilizing metadata for each process' data. EMPRESS provides a simple example of the next step in this evolution by integrating per-process metadata with thestorage system itself, making it more broadly useful than single file or application formats. Additionally, it allows for more robust and customizable metadata.