John Biddiscombe and Jerome Soumagne presented at the last HPC Advisory Council workshop in Lugano techniques for data staging for in-situ processing and parallel IO/Coupling of HPC applications:
Parallel IO is used by HPC applications to quickly export data to disk, but the time/space taken to save and reload the results for post-processing is often prohibitive and so data-staging is emerging as a necessary and convenient method of holding data temporarily so that reduction operations can take place using post-processing tools. In this talk we discuss staging methods/libraries and how they have been used at CSCS for coupling of simulation and analysis for computational steering, and the future developments planned to cope with the ever growing size of data produced by simulations.
