Overview

In 2020 the Data Intensive Research in Astrophysics and Cosmology institute (DIRAC), University of Washington (WA) Astronomy Department, Legacy Survey of Space and Time (LSST) and Amazon Web Services (AWS) joined forces to develop a proof-of-concept (PoC) leveraging cloud resources to processing astronomical images at scale. The AWS PoC’s goal was to implement required functionality that would enable Vera C. Rubin Observatory LSST Science Pipelines execution on AWS. The AWS PoC leveraged HTCondor Annex and Pegasus to successfully create and execute workflows in heterogeneous dynamically resizable compute clusters in order to determine feasibility, performance and provide initial cost estimates of such a system.

Rubin and LSST

Rubin Observatory is an 8m telescope currently under construction in Chile on which LSST sky survey will be conducted. The sky will be imaged by an incredible 3200 Megapixel camera capturing largest ever single-shot images. Because of it’s large field-of-view LSST will image the entire southern hemisphere of the sky every 3 nights over a period of 10 years.

Rubin estimates the size of nightly data products on the order of 20TB. Eventually, over the course of 10 years, it is estimated that Rubin will deliver 500 Petabytes (PB) of images and data products. How to process data and how to facilitate science at these data volumes are the two central questions of the Rubin Data Management (DM) team.

DM estimates that at the end of the ten-year survey they will use 950 TFLOPS (trillion floating point operations per second) to process all of the data and produce final Data Release 11. This processing is set to occur at two supercomputing facilities: SLAC National Accelerator Laboratory (SLAC) in US and CC-IN2P3 in France.

Around 10% of these resources will also be used to host services and provide data access to astronomers. Given the size of SLAC and CC-IN2P3 even the 10% of their resources is a significant amount of computational resources, but they are also not a one-size-fits-all solution. We wanted to investigate how we can give users access to state-of-the-art processing algorithms within Rubin DM data processing pipelines, can this be done in a way that scales to the size of Rubin and whether it can be affordable.

Middleware

Science Pipelines implement state of the art functionality in astronomical data reduction. They enable processing of data from a single exposure to joint calibration of multiple exposures for multiple different instruments, not just Rubin. The data processing algorithms are encapsulated in a concept known as a Task. A Task takes in a dataset as a Python object and performs some analysis on it, producing other datasets. Tasks can be chained in a pipeline represented by a directed acyclic graph (DAG) called a Quantum Graph (QG). Each node of a QG is a Task applied to an individual dataset.

Tasks operate on Python objects, the pairing of the Tasks and datasets are encoded in the Quantum Graph. The input-output (IO) operations are abstracted away into a component called the Data Butler. The Butler isolates users from the underlying file organization, file types and related file access mechanisms. Butler resolves a dataset ID through the Registry and retrieves the data from a Datastore. In simplified terms, the Datastore is the data storage, a POSIX filesystem for example, while the Registry, an SQL database, stores the provenance, metadata, IDs and other dataset relationships. By adding support for cloud services to the Butler we effectively gain the ability to run Science Pipelines in the cloud from a single compute instance – whether local or in the cloud.

Publications:  

D. Bektesevic, H. F. Chiang, et. al. A Gateway to Astronomical Image Processing: Vera C. RubinObservatory LSST Science Pipelines on AWS, https://arxiv.org/abs/2011.06044 , 2020

H.F. Chiang, D. Bektesevic, and the AWS-PoC team, “AWS Proof of Concept Project Report.” https://dmtn-137.lsst.io, 2020

H. F. Chiang, K. T. Liam, “Report of Google Cloud Proof of Concept” https://dmtn-157.lsst.io, 2020

H. F. Chiang, F. Huang et. al. Running Astronomical Pipelines with HTCondor, https://www.youtube.com/watch?v=CQgLEWZ_E1c

Contacts: 

Dino Bektešević (LSST, University of Washington)

H-F Chiang (NCSA)