Data Collection and Monitoring Across Heterogeneous Workflows in Pegasus

with No Comments

We will be holding regular online Pegasus Office Hours on Friday April 3rd, 2020 at 11AM Pacific.

During this installment of Pegasus Office hours, we will be presenting a data collection pipeline we have developed to enable researchers of the Laser Inferometer Gravitational-Wave Observatory (LIGO) to easily monitor and analyze complex scientific workflows.
First, we discuss the need for such data collection and monitoring solutions. This includes an introduction into our work: Domain-Aware Management of Heterogeneous Workflows: Active Data Management for Gravitational-Wave Science Workflows.
Following that will be an overview of the data collection pipeline itself along the components used: Elasticasearch, Logstash, Kibana, RabbitMQ, and Grafana.
Finally, we provide a containerized solution so that you can get up and running collecting data from your Pegasus workflow runs. A demonstration will be held covering the setup of this pipeline, configuring Pegasus to publish data, and  how to use the dashboard we have implemented.

To join the meeting on a computer or mobile phone:

Meeting ID: 491 318 546

One tap mobile
+16699006833,,491318546# US (San Jose)
+13462487799,,491318546# US (Houston)

Dial by your location
+1 669 900 6833 US (San Jose)
+1 346 248 7799 US (Houston)
+1 312 626 6799 US (Chicago)
+1 646 876 9923 US (New York)
+1 253 215 8782 US
+1 301 715 8592 US
Meeting ID: 491 318 546

See the Online Office Hours Series Page