Pegasus 4.9.0 Released

with No Comments

We are pleased to announce release of Pegasus 4.9.0

Pegasus 4.9.0 is be a major release of Pegasus. Highlights of new features:

  • Integrity Checking – Pegasus now performs integrity checking on files in a workflow for non shared filesystem deployments. More details can be found in the documentation at https://pegasus.isi.edu/documentation/integrity_checking.php . This work is the result of the Scientific Workflow Integration with Pegasus project (NSF awards 1642070, 1642053, and 1642090). More information on SWIP is available at https://cacr.iu.edu/projects/swip/
  • AWS Batch –  Pegasus now provides a way to execute horizontally clustered jobs on Amazon AWS Batch Service using a new command line tool pegasus-aws-batch. In other words, you can get Pegasus to cluster each level of your workflow into a bag of tasks and run those clustered jobs on Amazon Cloud using AWS Batch Service. InMore details can be found in documentation at https://pegasus.isi.edu/docs/4.9.0/aws-batch.php .
  • Pegasus Tutorial With Containers – We have a version of the tutorial for prospective new users to try out at  https://pegasus.isi.edu/tutorial/isi/ . This tutorial walks through users on how to bundle their application code in containers and use them for execution with Pegasus.

The release can be downloaded from
https://pegasus.isi.edu/downloads

Please note that the Perl DAX API is deprecated starting 4.9.0 Release and will be removed in the 5.0 Release.

Exhaustive list of features, improvements and bug fixes can be found below.

New Features

  • [PM-1179] – Integrity checking in Pegasus
  • [PM-1228] – Integrate Pegasus with AWS Batch
  • [PM-1167] – Dashboard should explain why condor submission failed
  • [PM-1233] – update pyopen ssl to 0.14 or higher
  • [PM-1265] – monitord to send events from job stdout
  • [PM-1277] – pegasus workflows on OSG should appear in OSG gratia
  • [PM-1280] – incorporate container based example in pegasus-init
  • [PM-1283] – tutorial vm should be updated to be able to run the containers tutorial
  • [PM-1289] – ability to filter events sent to AMQP endpoint
  • [PM-1298] – symlinking when running with containers
  • [PM-1310] – integrity checking documentation

Improvements

  • [PM-1288] – pegasus.project profile key is not set for SLURM submissions
  • [PM-898] – Modify monitord so that it can send AMQP messages AND populate a DB
  • [PM-1243] – Clusters of size 1 should be allowed when using AWS Batch and Horizontal Clustering
  • [PM-1244] – analyzer is not showing location of submit file
  • [PM-1248] – Planning fails in shared-fs mode with cryptic message.
  • [PM-1258] – Update AMQP support for Panorama
  • [PM-1270] – transformation selectors should preferred the site assigned by site selector
  • [PM-1285] – Exporting environmental variables in containers with the same technique
  • [PM-1294] – update flask dependency
  • [PM-1299] – Consider capturing which API was used to generate the DAX in metrics.
  • [PM-1308] – changes to database connection code
  • [PM-1313] – allow for direct use of singularity containers on CVMFS on compute site

Tasks

  • [PM-1284] – pegasus-mpi-cluster’s test PM848 fails on Debian 10
  • [PM-901] – site for OSG
  • [PM-1200] – update pegasus lite mode to support singulartiy
  • [PM-1231] – support pegasus-aws-batch as clustering executable
  • [PM-1246] – add manpage for pegasus-aws-batch
  • [PM-1250] – update handling for raw input files for integrity checking
  • [PM-1251] – pegasus-transfer to checksum files
  • [PM-1252] – enforce integrity checking for stage-out jobs
  • [PM-1254] – handling of checkpoint files for integrity checking
  • [PM-1257] – propagate checksums in hierarchal workflows
  • [PM-1260] – Integrity data in Stampede / statistics
  • [PM-1271] – update database schema and pegasus-db-admin to include integrity meta table
  • [PM-1272] – pegasus-integrity-check to log integrity data in monitord parseable event
  • [PM-1273] – pegasus-integrity-check to check for multiple files
  • [PM-1274] – update monitord to populate integrity events
  • [PM-1279] – Show integrity metrics in dashboard statistics page
  • [PM-1295] – pegasus should be able to report integrity errors
  • [PM-1296] – update database schema with tag table
  • [PM-1302] – pegasus-transfer should expose option for verify source existent for symlinks
  • [PM-1303] – update jupiter tc api for container mount keyword
  • [PM-1304] – pegasus-transfer should support docker->singularity pulls
  • [PM-1305] – Integrity checking breaks for symlinks in containers with different mount points
  • [PM-1306] – dials for integrity checking
  • [PM-1311] – Update DAX generators to generate workflow metadata key dax.api
  • [PM-1312] – track the dax_api key in the normalized database schema for metrics server

 

Bugs Fixed

  • [PM-1219] – Fix Foreign key constraint in workflow_files table
  • [PM-1221] – source tar balls have .git files
  • [PM-1222] – condor dagman does not allow . in job names
  • [PM-1223] – Python DAX API available via PIP?
  • [PM-1224] – sub workflow planning pegasus lite prescript does not associate credentials
  • [PM-1225] – hierarchal workflows planning in sharedfs fails with worker package staging set
  • [PM-1226] – hierarchal workflow with worker package staging fails
  • [PM-1234] – bucket creation for S3 as staging site fails
  • [PM-1245] – Blank space in remote_environment variable generates blank export command
  • [PM-1249] – Build fails against newer PostgreSQL
  • [PM-1253] – Planner should complain for same file designated as input and output
  • [PM-1255] – Singularity 2.4.(2?) pull cli has changed
  • [PM-1256] – rc-client does not strip quotes from PFN while populating
  • [PM-1261] – PMC .in files are not generated into the 00/00 pattern folder
  • [PM-1263] – Invalid raise statement in Python
  • [PM-1266] – Jupyter API does not only plans the workflow without submitting it
  • [PM-1275] – kickstart build fails on arm64
  • [PM-1276] – unbounded recursion in database loader for dashboard in monitord
  • [PM-1281] – pegasus-analyzer does not show task stdout/stderr for held jobs
  • [PM-1282] – pegasus.runtime doesn’t affect walltime
  • [PM-1287] – python3 added as a rpm dep, even though it is not a true dep
  • [PM-1290] – Queue is not handled correctly in Glite
  • [PM-1291] – DB integrity errors in 4.9.0dev
  • [PM-1292] – incomplete kickstart output
  • [PM-1293] – stage-in jobs always have generate_checksum set to true if checksums are not present in RC
  • [PM-1297] – monitord does not prescript log in case of planning failure for sub workflows
  • [PM-1300] – Planning error when image is of type singularity and source is docker hub
  • [PM-1301] – Python exception raised when using MySQL for stampede DB
  • [PM-1307] – some of the newly generated events are missing timestamp
  • [PM-1314] – incorrect way of computing suffix for singularity images
  • [PM-1315] – Exception when data reuse removes all jobs.
  • [PM-1316] – Transformation that uses a regular file, but defines it in RC raises an exception, unlike in older versions
  • [PM-1317] – Pegasus plan fails on container task clustering

 

174 views