• Workflow Level in the DAX
• Task level in the DAX and the Transformation Catalog
• File level in the DAX and Replica Catalog

Metadata is specified as a key value tuple, where both key and values are of type String.

All the metadata ( user specified and auto-generated) gets populated into the workflow database ( usually in the workflow submit directory) by pegasus-monitord. The metadata in this database can be be queried for using the pegasus-metadata command line tool, or is also shown in the Pegasus Dashboard.

Documentation: https://pegasus.isi.edu/wms/docs/4.6.0/metadata.php

Relevant JIRA items

• [PM-917] – modify the workflow database to associate metadata with #1035 workflow, job and files
• [PM-918] – modify pegasus-monitord to populate metadata into #1036 stampede database
• [PM-919] – pegasus-metadata command line tool #1037
• [PM-916] – identify and generate the BP events for metadata #1034
• [PM-913] – kickstart support for stat command line options #1031
• [PM-1025] – Document the metadata capability for 4.6 #1140
• [PM-992] – automatically capture file metadata from kickstart and record it #1109
• [PM-892] – Add metadata to DAX schema #1010
• [PM-893] – Add metadata to Python DAX API #1011
• [PM-894] – Add metadata to site catalog schema #1012
• [PM-895] – Add metadata to transformation catalog text format #1013
• [PM-902] – support for metadata to JAVA DAX API #1020
• [PM-903] – add metadata to perl dax api #1021
• [PM-904] – support for parsing DAX 3.6 documents #1022
• [PM-978] – Update JDBCRC with the new schema #1095
• [PM-925] – support for 4.1 new site catalog schema with metadata extensions #1043
• [PM-991] – pegasus dashboard to display metadata stored in workflow database #1108

Pegasus Planner supports notion of variable expansions in the DAX and the catalog files along the same lines as bash variable expansion works. This is often useful, when you want paths in your catalogs or profile values in the DAX to be picked up from the environment. An error is thrown if a variable cannot be expanded.

Variable substitution is supported in the DAX, File Based Replica Catalog, Transformation Catalog and the Site Catalog.

Documentation: https://pegasus.isi.edu/wms/docs/4.6.0/variable_expansion.php

Relevant JIRA items

• [PM-831] – Add better support for variables #949

The planner now has support for a new cleanup algorithm called constraint. The algoirthm adds cleanup nodes to constraint the amount of storage space used by a workflow. The nodes remove files no longer required during execution. The added cleanup node guarantees limits on disk usage. The leaf cleanup nodes are also added when this is selected.

• [PM-850] – Integrate Sudarshan’s cleanup algorithm #968

Users can now specify Pegasus profiles to indicate resource requirements for jobs. Pegasus will automatically, translate these to the approprate condor, globus or batch system keys based on how the job is executed.

The profiles are documented in the configuration chapter at ask requirement profiles are documented here https://pegasus.isi.edu/wms/docs/4.6.0/profiles.php#pegasus_profiles

• [PM-962] – common pegasus profiles to indicate resource requirements for job #1079

A new command line client called “pegasus-init” that generates a new workflow configuration based by asking the user a series of questions. Based on the responses to these questions,pegasus-init generates a workflow configuration including a DAX generator, site catalog, properties file, and other artifacts that can be edited to meet the user’s needs.

• [PM-1019] – pegasus-init client to setup pegasus on a machine #1134

Pegasus, now during Replica Selection orders all the candidate replica’s instead of selecting the best replica. This replica’s are ordered based on the strategy selected, and the ordered list is passed to pegasus-transfer invocation. This allows users to specify failover, or preferred location for discovering the input files.

By default, planner employs the following logic for the ordering of replicas

• valid file URL’s . That is URL’s that have the site attribute matching the site where the executable pegasus-transfer is executed.
• all URL’s from preferred site (usually the compute site)
• all other remotely accessible ( non file) URL’s

If a user wants to specify their own order preference , then they should use the Regex Replica Selector and specify a ranked order list of regular expressions in the properties.

Documentation: https://pegasus.isi.edu/wms/docs/4.6.0/data_management.php#replica_selection

Relevant JIRA items:

• [PM-1002] – Support symlinking against compute site datasets in #1119 nonsharedfs mode with bypass of input file staging
• [PM-1014] – Support for Fallback PFN while transferring raw input files #1130

Pegasus now has support for submitting to a local SGE cluster via the HTCondor Glite/Blahp interfaces. More details can be found in the documentation at https://pegasus.isi.edu/wms/docs/4.6.0/glite.php

• [PM-955] – Support for direct submission through SGE using #1072 Condor/Glite/Blahp layer

Users don’t need to set extra pegasus profiles to enable jobs to run correctly on glite style sites. By default, condor quoting for jobs on glite style sites is disabled. Also, the -w option to kickstart is always as batch gahp does not support specification of a remote execution directory directly.

If the user knows that a compute site shares a file system with the submit host, then they can get Pegasus to run the auxillary jobs in local universe. This is especially helpful , when submitting to local campus clusters using Glite and users don’t want the pegasus auxillary jobs to run through the cluster PBS|SGE queue.

Relevant JIRA items

• [PM-934] – changed how environment is set for jobs submitted via #1051 HTCondor Glite / Blahp layer
• [PM-1024] – Use local universe for auxiliary jobs in glite/blahp mode #1139
• [PM-1037] – Disable Condor Quoting for jobs run on glite style #1151 execution sites
• [PM-960] – Set default working dir to scratch dir for glite style jobs #1077

• [PM-967] – Add support for PAPI CPU counters in Kickstart #1084

Pegasus now by default, attempts to use the worker package out of the Pegasus submit host installation unless a user has specified finer grained attributes for the compute sites in the site catalog or an entry is specified in the transformation catalog.

Relevant JIRA items

• [PM-888] – Guess which worker package to use based on the submit host #1006

Users can specify a different output replica catalog optionally by specifying the property with prefix pegasus.catalog.replica.output

This is useful when users want to separate the replica catalog that they use for discovery of input files and the catalog where the output files generated are registered. For example use a Directory backed replica catalog backend to discover file locations, and a file based replica catalog to catalog the locations of the output files.

pegasus-transfer now batches 70 transfers in a single scp invocation against the same host.

scp rm are now batched together at a level of 70 per group so that we can keep the command lines short enough.

s3 removes are now batched and passed in a temp file to pegasus-s3

Kickstart supports an environment variable, KICKSTART_WRAPPER that contains a set of command-line arguments to insert between Kickstart and the application

When a user condor_rm’s their job, Condor sends the job a SIGTERM. Previously this would cause Kickstart to die. This commit changes Kickstart so that it catches the SIGTERM and passes it on to the child instead. That way the child dies, but not Kickstart, and Kickstart can report an invocation record forthe job to provide the user with useful debugging info. This same logic is also applied to SIGINT and SIGQUIT.

pegasus will default the –conf option to pegasus-plan to pegasus.properties in the current working directory. In addition, the default locations for the various catalog files now point to current working directory ( rc.txt, tc.txt, sites.xml )