Pegasus 4.9.0 Released

by on October 31, 2018 with No Comments

We are pleased to announce release of Pegasus 4.9.0

Pegasus 4.9.0 is be a major release of Pegasus. Highlights of new features:

  • Integrity Checking – Pegasus now performs integrity checking on files in a workflow for non shared filesystem deployments. More details can be found in the documentation at https://pegasus.isi.edu/documentation/integrity_checking.php . This work is the result of the Scientific Workflow Integration with Pegasus project (NSF awards 1642070, 1642053, and 1642090). More information on SWIP is available at https://cacr.iu.edu/projects/swip/
  • AWS Batch –  Pegasus now provides a way to execute horizontally clustered jobs on Amazon AWS Batch Service using a new command line tool pegasus-aws-batch. In other words, you can get Pegasus to cluster each level of your workflow into a bag of tasks and run those clustered jobs on Amazon Cloud using AWS Batch Service. InMore details can be found in documentation at https://pegasus.isi.edu/docs/4.9.0/aws-batch.php .
  • Pegasus Tutorial With Containers – We have a version of the tutorial for prospective new users to try out at  https://pegasus.isi.edu/tutorial/isi/ . This tutorial walks through users on how to bundle their application code in containers and use them for execution with Pegasus.

The release can be downloaded from
https://pegasus.isi.edu/downloads

Please note that the Perl DAX API is deprecated starting 4.9.0 Release and will be removed in the 5.0 Release.

Exhaustive list of features, improvements and bug fixes can be found below.

New Features

  1. [PM-1179] – Integrity checking in Pegasus
  2. [PM-1228] – Integrate Pegasus with AWS Batch
  3. [PM-1167] – Dashboard should explain why condor submission failed
  4. [PM-1233] – update pyopen ssl to 0.14 or higher
  5. [PM-1265] – monitord to send events from job stdout
  6. [PM-1277] – pegasus workflows on OSG should appear in OSG gratia
  7. [PM-1280] – incorporate container based example in pegasus-init
  8. [PM-1283] – tutorial vm should be updated to be able to run the containers tutorial
  9. [PM-1289] – ability to filter events sent to AMQP endpoint
  10. [PM-1298] – symlinking when running with containers
  11. [PM-1310] – integrity checking documentation

Improvements

  1. [PM-1288] – pegasus.project profile key is not set for SLURM submissions
  2. [PM-898] – Modify monitord so that it can send AMQP messages AND populate a DB
  3. [PM-1243] – Clusters of size 1 should be allowed when using AWS Batch and Horizontal Clustering
  4. [PM-1244] – analyzer is not showing location of submit file
  5. [PM-1248] – Planning fails in shared-fs mode with cryptic message.
  6. [PM-1258] – Update AMQP support for Panorama
  7. [PM-1270] – transformation selectors should preferred the site assigned by site selector
  8. [PM-1285] – Exporting environmental variables in containers with the same technique
  9. [PM-1294] – update flask dependency
  10. [PM-1299] – Consider capturing which API was used to generate the DAX in metrics.
  11. [PM-1308] – changes to database connection code
  12. [PM-1313] – allow for direct use of singularity containers on CVMFS on compute site

Tasks

  1. [PM-1284] – pegasus-mpi-cluster’s test PM848 fails on Debian 10
  2. [PM-901] – site for OSG
  3. [PM-1200] – update pegasus lite mode to support singulartiy
  4. [PM-1231] – support pegasus-aws-batch as clustering executable
  5. [PM-1246] – add manpage for pegasus-aws-batch
  6. [PM-1250] – update handling for raw input files for integrity checking
  7. [PM-1251] – pegasus-transfer to checksum files
  8. [PM-1252] – enforce integrity checking for stage-out jobs
  9. [PM-1254] – handling of checkpoint files for integrity checking
  10. [PM-1257] – propagate checksums in hierarchal workflows
  11. [PM-1260] – Integrity data in Stampede / statistics
  12. [PM-1271] – update database schema and pegasus-db-admin to include integrity meta table
  13. [PM-1272] – pegasus-integrity-check to log integrity data in monitord parseable event
  14. [PM-1273] – pegasus-integrity-check to check for multiple files
  15. [PM-1274] – update monitord to populate integrity events
  16. [PM-1279] – Show integrity metrics in dashboard statistics page
  17. [PM-1295] – pegasus should be able to report integrity errors
  18. [PM-1296] – update database schema with tag table
  19. [PM-1302] – pegasus-transfer should expose option for verify source existent for symlinks
  20. [PM-1303] – update jupiter tc api for container mount keyword
  21. [PM-1304] – pegasus-transfer should support docker->singularity pulls
  22. [PM-1305] – Integrity checking breaks for symlinks in containers with different mount points
  23. [PM-1306] – dials for integrity checking
  24. [PM-1311] – Update DAX generators to generate workflow metadata key dax.api
  25. [PM-1312] – track the dax_api key in the normalized database schema for metrics server

Bugs Fixed

  1. [PM-1219] – Fix Foreign key constraint in workflow_files table
  2. [PM-1221] – source tar balls have .git files
  3. [PM-1222] – condor dagman does not allow . in job names
  4. [PM-1223] – Python DAX API available via PIP?
  5. [PM-1224] – sub workflow planning pegasus lite prescript does not associate credentials
  6. [PM-1225] – hierarchal workflows planning in sharedfs fails with worker package staging set
  7. [PM-1226] – hierarchal workflow with worker package staging fails
  8. [PM-1234] – bucket creation for S3 as staging site fails
  9. [PM-1245] – Blank space in remote_environment variable generates blank export command
  10. [PM-1249] – Build fails against newer PostgreSQL
  11. [PM-1253] – Planner should complain for same file designated as input and output
  12. [PM-1255] – Singularity 2.4.(2?) pull cli has changed
  13. [PM-1256] – rc-client does not strip quotes from PFN while populating
  14. [PM-1261] – PMC .in files are not generated into the 00/00 pattern folder
  15. [PM-1263] – Invalid raise statement in Python
  16. [PM-1266] – Jupyter API does not only plans the workflow without submitting it
  17. [PM-1275] – kickstart build fails on arm64
  18. [PM-1276] – unbounded recursion in database loader for dashboard in monitord
  19. [PM-1281] – pegasus-analyzer does not show task stdout/stderr for held jobs
  20. [PM-1282] – pegasus.runtime doesn’t affect walltime
  21. [PM-1287] – python3 added as a rpm dep, even though it is not a true dep
  22. [PM-1290] – Queue is not handled correctly in Glite
  23. [PM-1291] – DB integrity errors in 4.9.0dev
  24. [PM-1292] – incomplete kickstart output
  25. [PM-1293] – stage-in jobs always have generate_checksum set to true if checksums are not present in RC
  26. [PM-1297] – monitord does not prescript log in case of planning failure for sub workflows
  27. [PM-1300] – Planning error when image is of type singularity and source is docker hub
  28. [PM-1301] – Python exception raised when using MySQL for stampede DB
  29. [PM-1307] – some of the newly generated events are missing timestamp
  30. [PM-1314] – incorrect way of computing suffix for singularity images
  31. [PM-1315] – Exception when data reuse removes all jobs.
  32. [PM-1316] – Transformation that uses a regular file, but defines it in RC raises an exception, unlike in older versions
  33. [PM-1317] – Pegasus plan fails on container task clustering

4,860 views