We are pleased to announce a beta release Pegasus 4.8.0beta3 for upcoming Pegasus 4.8.0.

Pegasus 4.8.0 will be a major release of Pegasus and includes improvements and bug fixes to the 4.7.4 release.

Pegasus 4.8.0 Release has support for

• Application Containers – Pegasus now supports containers for user applications. Both Docker and Singularity are supported. More details can be found in the documentation at https://pegasus.isi.edu/docs/4.8.0dev/containers.php
• Jupyter Support –  Pegasus now provides a Python API to declare and manage workflows via Jupyter, which allows workflow creation, execution, and monitoring. The API also provides mechanisms to create Pegasus catalogs (sites, replica, and transformation). More details can be found in the documentation at https://pegasus.isi.edu/docs/4.8.0dev/jupyter.php
• Tuning of Transfer and Cleanup jobs – Pegasus now computes the number of transfer and cleanup jobs to be added for a workflow for a particular level, according to number of jobs on that level https://pegasus.isi.edu/docs/4.8.0dev/data_transfers.php

The beta release can be downloaded from

https://download.pegasus.isi.edu/pegasus/4.8.0beta3/

Exhaustive list of features, improvements and bug fixes can be found below

New Features and Improvements

• [PM-1134] – Capture the execution site information in pegasus lite
• [PM-1159] – Support for containers
• [PM-1177] – API for running Pegasus workflows via Jupiter

• [PM-1109] – dashboard to display errors if a job is killed instead of exiting with non zero exitcode
• [PM-1146] – There doesn’t seem to be a way to get a persistent URL for a workflow in dashboard
• [PM-1155] – remote cleanup jobs should have file url’s if possible
• [PM-1158] – Make DAX3 API compatible with Python 2.6+ and Python3+
• [PM-1161] – Update documentation of large databases for handling mysqldump: Error 2013
• [PM-1187] – make scheduler type case insensitive for grid gateway in site catalog
• [PM-1212] – new defaults for number of transfer and inplace jobs created

• [PM-1165] – Update Transformation Catalog format to support containers
• [PM-1166] – pegasus-transfer to support transfers from docker hub
• [PM-1180] – update monitord to populate checksums
• [PM-1183] – monitord plumbing for putting hooks for integrity checking
• [PM-1188] – Add tool to integrity check transferred files
• [PM-1190] – planner changes to enable integrity checking
• [PM-1194] – update planner pegasus lite mode to support for docker container wrapper
• [PM-1195] – update site selection to handle containers
• [PM-1197] – handle symlinks for input files when launching job via container
• [PM-1200] – update pegasus lite mode to support singulartiy
• [PM-1201] – Transformation Catalog API should support the container keywords
• [PM-1202] – Move catalog APIs into Pegasus.catalogs and develop standalone test cases independent from Jupyter
• [PM-1210] – update pegasus-transfer to support transfers from singularity hub
• [PM-1214] – Specifying environment for sites and containers
• [PM-1215] – Document support for containers for 4.8

 

Bugs Fixed

• [PM-1032] – Handle proper error message for non-standard python usage
• [PM-1131] – stage in jobs repeat portions of deep LFN’s
• [PM-1132] – Hashed staging mapper doen’t work correctly with sub dax generation jobs
• [PM-1135] – pegasus.transfer.bypass.input.staging breaks symlinking on the local site
• [PM-1136] – With bypass input staging some URLs are ending up in the wrong site
• [PM-1141] – The commit to allow symlinks in pegasus-transfer broke PFN fall through
• [PM-1142] – Do not set LD_LIBRARY_PATH in job env
• [PM-1144] – pegasus lite prints the wrong hostname for non-glidein jobs
• [PM-1147] – pegasus-transfer should check that files exist before trying to transfer them
• [PM-1148] – kickstart should print a more helpful error message if the executable is missing
• [PM-1151] – pegasus-monitord fails to populate stampede DB correctly when workflow is run on HTCondor 8.5.8
• [PM-1152] – pegasus-analyzer not showing stdout and stderr of failed transfer jobs
• [PM-1153] – Pegasus creates extraneous spaces when replacing <file name=”something” />
• [PM-1154] – regex too narrow for GO names with dashes
• [PM-1157] – monitord replay should work on submit directories that are moved
• [PM-1160] – Dashboard is not recording the hostname correctly
• [PM-1162] – Running pegasus-monitord replay created an unreadable database
• [PM-1163] – Confusing error message in pegasus-kickstart
• [PM-1164] – worker package in submit directory gets deleted during workflow run
• [PM-1171] – Monitord regularly produces empty stderr and stdout files
• [PM-1172] – pegasus-rc-client deletes all entries for a lfn
• [PM-1173] – cleanup jobs failing against Titan gridftp server due to RFC 2818 compliance
• [PM-1174] – monitord should maintain the permissions on ~/.pegasus/workflow.db
• [PM-1176] – the job notifications on failure and success should have exitcode from kickstart file
• [PM-1181] – monitord fails to exit if database is locked
• [PM-1182] – registration jobs fail if a file based RC has variables defined
• [PM-1185] – destination in remote file transfers for inter site jobs point’s to directory
• [PM-1189] – Making X86_64 the default arch in the site catalog
• [PM-1191] – If available, use GFAL over guc
• [PM-1192] – User supplied env setup script for lite
• [PM-1193] – “pegasus-rc-client list” modifies rc.txt
• [PM-1196] – pegasus-statistics is not generating jobs.txt for some large workflows
• [PM-1207] – Investigate error message: Normalizing ‘4.8.0dev’ to ‘4.8.0.dev0’
• [PM-1208] – Improve database is locked error message
• [PM-1209] – Analyzer gets confused about retry number in hierarchical workflows
• [PM-1211] – DAX API should tell which lfn was a dup
• [PM-1213] – pegasus creates duplicate source URL’s for staged executables
• [PM-1217] – monitord exits prematurely, when in dagman recovery mode

Technical task

• [PM-1178] – kickstart to checksum output files

 

3,439 views