Reworked Python API:

This new API has been developed from the ground up so that, in addition to generating the abstract workflow and all the catalogs, it now allows you to plan, submit, monitor, analyze and generate statistics of your workflow. To use this new Python API refer to the Moving From DAX3 to Pegasus.api at https://pegasus.isi.edu/docs/5.0.0/user-guide/migration.html#moving-from-dax3

Adoption of YAML formats:

With Pegasus 5.0, we are moving to adoption of YAML for representation of all major catalogs. We have provided catalog converters for you to convert your existing catalogs to the new formats. In 5.0, the following are now represented in YAML:

• Abstract Workflow

• Replica Catalog

• Transformation Catalog

• Site Catalog

• Kickstart Provenance Records

Python3 Support:

All Pegasus tools are Python 3 compliant. 5.0 release will require Python 3 on workflow submit node Python PIP packages for workflow composition and monitoring

Default data configuration

In Pegasus 5.0, the default data configuration has been changed to condorio . Up to 4.9.x releases, the default configuration was sharedfs.

Zero configuration required to submit to local HTCondor pool

Data Management Improvements

• New output replica catalog that registers outputs including file metadata such as size and checksums

• Ability to do bypass staging of files at a per file, executable and container level

• Improved support for hierarchal workflows allow you to create data dependencies between sub workflow jobs and compute jobs

• Support for staging of generated outputs to multiple output sites

• Support for integrity checking of user executables and application containers in addition to data

• Support for webdav transfers

• Easier enabling of data reuse by specifying previous workflow submit directories using –reuse option to pegasus-plan.

• Stagein transfer jobs are assigned priorities based on the number of child compute jobs. Details can be found in JIRA ticket 1385

New Jupyter Notebook Based Tutorial

With this release, we are pleased to announce a brand new tutorial based on a Docker container running interactive Jupyter notebooks. You can access the tutorial at PM-1385 #1499

Support for CWL (Common Workflow Language)

The pegasus-cwl-converter command line tool has been developed to convert a subset of the Common Workflow Language (CWL) to Pegasus’s native YAML format. Given the following three files: a CWL workflow file, a workflow inputs specification file, and a transformation (executable) specification file, pegasus-cwl-converter will do a best-effort translation. This will also work with CWL workflow specifications that refer to Docker containers. The entire CWL language specification is not yet covered by this converter, and as such we can provide additional support in converting your workflows into the Pegasus YAML format.

Support for triggers in Ensemble Manager

The Pegasus Ensemble Manager is a service that manages collections of workflows. In this latest release of Pegasus, workflow triggering functionality has been added to this service. With the ensemble manager service up and running, the pegasus-em command can now be used to start workflow triggers. Two triggersare currently supported:

• a cron based trigger: The cron based trigger will, at a given interval, submit a new workflow to your ensemble.

• a cron based, file pattern trigger: The cron based, file pattern trigger, much like the cron based trigger, will submit a new workflow to your ensemble at a given time interval, with the addition of any new files that are detected based on a given file pattern. This is useful for automatically processing data as it arrives.

Improved events for each job reported to AMQP

Historically the events reported to AMQP endpoints are normalized events corresponding to the stampede database, which makes correlation hard. Pegasus now also reports a new job composite event (stampede.job_inst.composite) to AMQP end points that have a complete information about a job execution.

Revamped Documentation

Documentation has been overhauled and broken down into a user guide and a reference guide. In addition, we have moved to readthedocs style documentation using restructured text. The documentation can be found at https://pegasus.isi.edu/docs/5.0.0/index.html .

pegasus-statistics reports memory usage and avg cpu utilization

pegasus-statistics now reports memory usage and average cpu utilization for your jobs in the transformation statistics file (breakdown.txt).

PegasusLite Improvements

• Users can now specify environment setup scripts in the site catalog that need to be sourced to setup the environment before the job is launched by PegasusLite. More details at https://pegasus.isi.edu/docs/5.0.0/reference-guide/pegasus-lite.html#setting-the-environment-in-pegasuslite-for-your-job

• Users can get the transfers in PegasusLite to run on DTN nodes while the jobs run on the the compute nodes. More details at https://pegasus.isi.edu/docs/5.0.0/reference-guide/pegasus-lite.html#specify-compute-job-in-pegasuslite-to-run-on-different-node

Credentials existence is checked upfront by planner

Performance improvements for pegasus-rc-client

pegasus-rc-client now does bulk inserts when inserting entries into a database backed Replica Catalog.

Pegasus ensures a consistent UTF8 environment across full workflow Details can be found in JIRA ticket 1592 .