Pegasus 4.8.0 Released

by on September 5, 2017 with No Comments

We are pleased to announce release of Pegasus 4.8.0 

Pegasus 4.8.0 is be a major release of Pegasus and includes improvements and bug fixes to the 4.7.4 release.

Pegasus 4.8.0 Release has support for

  • Application Containers – Pegasus now supports containers for user applications. Both Docker and Singularity are supported. More details can be found in the documentation at https://pegasus.isi.edu/docs/4.8.0/containers.php
  • Jupyter Support –  Pegasus now provides a Python API to declare and manage workflows via Jupyter, which allows workflow creation, execution, and monitoring. The API also provides mechanisms to create Pegasus catalogs (sites, replica, and transformation). More details can be found in the documentation at https://pegasus.isi.edu/docs/4.8.0/jupyter.php 
  • Tuning of Transfer and Cleanup jobs – Pegasus now computes the number of transfer and cleanup jobs to be added for a workflow for a particular level, according to number of jobs on that level https://pegasus.isi.edu/docs/4.8.0/data_transfers.php

The release can be downloaded from
http://pegasus.isi.edu/downloads

Exhaustive list of features, improvements and bug fixes can be found below

New Features and Improvements

  • [PM-1159] – Support for containers
  • [PM-1177] – API for running Pegasus workflows via Jupyter
  • [PM-1191] – If available, use GFAL over globus url copy
    • JGlobus is no longer actively supported and is not in compliance with RFC 2818(https://docs.globus.org/security-bulletins/2015-12-strict-mode). As a result cleanup jobs using pegasus-gridftp client would fail against the servers supporting the strict mode. We have removed the pegasus-gridftp client and now use gfal clients as globus-url-copy does not support removes. If gfal is not available, globus-url-copy is used for cleanup by writing out zero bytes files instead of removing them.
  • [PM-1212] – new defaults for number of transfer and inplace jobs created
  • [PM-1134] – Capture the execution site information in pegasus lite
  • [PM-1109] – dashboard to display errors if a job is killed instead of exiting with non zero exitcode
  • [PM-1146] – There doesn’t seem to be a way to get a persistent URL for a workflow in dashboard
  • [PM-1155] – remote cleanup jobs should have file url’s if possible
  • [PM-1158] – Make DAX3 API compatible with Python 2.6+ and Python3+
  • [PM-1161] – Update documentation of large databases for handling mysqldump: Error 2013
  • [PM-1187] – make scheduler type case insensitive for grid gateway in site catalog
  • [PM-1165] – Update Transformation Catalog format to support containers
  • [PM-1166] – pegasus-transfer to support transfers from docker hub
  • [PM-1180] – update monitord to populate checksums
  • [PM-1183] – monitord plumbing for putting hooks for integrity checking
  • [PM-1188] – Add tool to integrity check transferred files
  • [PM-1190] – planner changes to enable integrity checking
  • [PM-1194] – update planner pegasus lite mode to support for docker container wrapper
  • [PM-1195] – update site selection to handle containers
  • [PM-1197] – handle symlinks for input files when launching job via container
  • [PM-1200] – update pegasus lite mode to support singulartiy
  • [PM-1201] – Transformation Catalog API should support the container keywords
  • [PM-1202] – Move catalog APIs into Pegasus.catalogs and develop standalone test cases independent from Jupyter
  • [PM-1210] – update pegasus-transfer to support transfers from singularity hub
  • [PM-1214] – Specifying environment for sites and containers
  • [PM-1215] – Document support for containers for 4.8
  • [PM-1178] – kickstart to checksum output files
  • [PM-1220] – default app name for metrics server based on dax label
  • [PM-1146] – There doesn’t seem to be a way to get a persistent URL for a workflow in dashboard
  • [PM-1186] – pegasus-db-admin should list compatibility with latest pegasus version if no changes to schema
  • [PM-1187] – make scheduler type case insensitive for grid gateway in site catalog

 

Bugs Fixed

  • [PM-1032] – Handle proper error message for non-standard python usage
  • [PM-1162] – Running pegasus-monitord replay created an unreadable database
  • [PM-1171] – Monitord regularly produces empty stderr and stdout files
  • [PM-1172] – pegasus-rc-client deletes all entries for a lfn
  • [PM-1173] – cleanup jobs failing against Titan gridftp server due to RFC 2818 compliance
  • [PM-1174] – monitord should maintain the permissions on ~/.pegasus/workflow.db
  • [PM-1176] – the job notifications on failure and success should have exitcode from kickstart file
  • [PM-1181] – monitord fails to exit if database is locked
  • [PM-1185] – destination in remote file transfers for inter site jobs point’s to directory
  • [PM-1189] – Making X86_64 the default arch in the site catalog
  • [PM-1193] – “pegasus-rc-client list” modifies rc.txt
  • [PM-1196] – pegasus-statistics is not generating jobs.txt for some large workflows
  • [PM-1207] – Investigate error message: Normalizing ‘4.8.0dev’ to ‘4.8.0.dev0’
  • [PM-1208] – Improve database is locked error message
  • [PM-1209] – Analyzer gets confused about retry number in hierarchical workflows
  • [PM-1211] – DAX API should tell which lfn was a dup
  • [PM-1213] – pegasus creates duplicate source URL’s for staged executables
  • [PM-1217] – monitord exits prematurely, when in dagman recovery mode
  • [PM-1218] – monitord replay against mysql with registration jobs 

5,753 views