Southern California Earthquake Center (SCEC) is a community of over 600 scientists, students, and others at over 60 institutions worldwide, headquartered at the University of Southern California. SCEC is funded by the National Science Foundation and the U.S. Geological Survey to develop a comprehensive understanding of earthquakes in Southern California and elsewhere, and to communicate useful knowledge for reducing earthquake risk. They use Pegasus in several of their scientific workflows using advanced features like Clustering, Recursive Daxes, Throttling, Priorities
CyberShake
Hazard maps generated as part of the SCEC CyberShake project, indicate the maximum amount of shaking expected at a particular geographic location over a certain period of time. The hazard maps are used by civil engineers to determine building design tolerances. CyberShake delivers new insights into how rupture directivity and sedimentary basin effects contribute to the shaking experienced at different geographic locations. As a result more accurate hazard maps can be created.
The CyberShake Workflows consists of MPI style (tightly coupled) jobs generating the Strain Green Tensors that are then postprocessed for each geographical site ( a 10x10km region). The post processing jobs are hundreds of thousands of loosely coupled jobs that generate seismograms and peak spectral acceleration values. Processed data from each site is used to generate a hazard curve and several hundred curves are used to create a hazard map.
Pegasus maps the CyberShake workflows onto SCEC, NSF XSEDE and OSG (Open Science Grid) CyberInfrastructure resources. The Hazard map on the right is the result of running CyberShake on SCEC and TeraGrid in the Spring of 2009. The workflows ran over a period of 56 days and produced about 276 TB of data (11TB final output + 265TB intermediate) using 607 CPU Years.
Several advanced features of Pegasus are being used to optimize the run time of the CyberShake workflows as well as to make them run reliably. The features used are Job Clustering, Recursive Daxes, Throttling, Priorities, Glideins, Job Retry, Workflow Retry, Data Clustering
BroadBand
Broadband platform enables researchers to combine long period (<1.0Hz) deterministic seismograms with high frequency (~10Hz) stochastic seismograms. These broadband seismograms contain the frequencies of interest to engineering community. Traditionally, earthquake engineers have used scalar ground motion intensity measures (i.e., spectral acceleration – SA, peak ground acceleration – PGA, peak ground velocity -PGV) obtained from ground motion prediction equations (GMPEs) for their building design procedures. In recent years, an increasing awareness has emerged of the advantages of using ground-motion time series for more realistic non-linear dynamic analysis of buildings and performance-based earthquake engineering.
SCEC plans to utilize Pegasus for running simulations in the Broadband platform. The image shows a broadband workflow for a single source, station and velocity file. Plans are to run workflows that involve 10’s to 100’s of sources and 100’s of stations.
Runs on National Cyberinfrastructure
As a result more accurate hazard maps can be created. In 2006, SCEC ran two CyberShake post-processing workflows on the TeraGrid. Those two workflows took 23 days to complete and included 87,000 jobs.
In 2008, SCEC increased the size of the workflows to over 840,000 tasks and increased the number of workflows to 36. The runtime of each of these workflows was 18 hrs using 400 cores. Substantial improvements in performance were realized through algorithmic changes in the application and the use of task clustering and other Pegasus optimizations. In 2009 SCEC ran 223 workflows over a period of eight weeks, with a workflow being completed every 5.4 hours. Over the eight weeks, an average 4,420 cores on TG Ranger was used at any point in time, with a peak of 14,540 cores. There were 192 million application tasks executed and 192 million files produced. A total of 11 TB of output data was stored and 165 TB of temporary data was generated. In 2010 CyberShake was also run across the National CyberInfrastructure (Open Science Grid and TeraGrid). SCEC also uses Pegasus to run the Broadband application, which generates and compares seismograms from several different earthquake simulation codes. The image below shows a broadband workflow for a single source, stations and velocity file. Plans are to run workflows that involve 10’s to 100’s of sources and 100’s of stations.
SCEC is also using Pegasus and DAGMan in the Earthworks Portal, a TeraGrid Science Gateway, hosted at Washington University that allows users to configure and execute earthquake wave propagation simulations structured as workflows through a simple portal interface.
David Okaya of SCEC has an interesting slide on Benefits of Scientific Workflows for an application scientists perspective.
Publications:
Scott Callaghan, Ewa Deelman, Dan Gunter, Gideon Juve, Philip Maechling, Christopher Brooks, Karan Vahi, Kevin Milner, Robert Graves, Edward Field, David Okaya, Thomas Jordan. Scaling up Workflow-based Applications , Journal of Computer and System Sciences. Available online 22 November 2009
Scott Callaghan, Philip Maechling, Ewa Deelman, Karan Vahi, Gaurang Mehta, Gideon Juve, Kevin Milner, Robert Graves, Edward Field, David Okaya, Dan Gunter, Keith Beattie, Thomas Jordan, Reducing Time-to-Solution Using Distributed High-Throughput Mega-Workflows – Experiences from SCEC CyberShake, Fourth IEEE International Conference on e-Science (e-Science 2008), 10-12 December 2008 in Indianapolis, Indiana, USA
Ewa Deelman, Scott Callaghan, Edward Field, Hunter Francoeur, Robert Graves, Nitin Gupta, Vipin Gupta, Thomas H. Jordan, Carl Kesselman, Philip Maechling, John Mehringer, Gaurang Mehta, David Okaya, Karan Vahi, Li Zhao. “Managing Large-Scale Workflow Execution from Resource Provisioning to Provenance tracking: The CyberShake Example“, e-Science 2006, Amsterdam, December 4-6, 2006
Philip Maechling, Hans Chalupsky, Maureen Dougherty, Ewa Deelman, Yolanda Gil, Sridhar Gullapalli, Vipin Gupta, Carl Kesselman, Jihie Kim, Gaurang Mehta, Brian Mendenhall, Thomas A. Russ, Gurmeet Singh, Marc Spraragen, Garrick Staples, Karan Vahi. “Simplifying construction of complex workflows for non-expert users of the Southern California Earthquake Center Community Modeling Environment” SIGMOD Record 34(3): 24-30 (2005)