Pegasus 2.2.0 Released

===============================
Release Notes for PEGASUS 2.2.0
===============================

NEW FEATURES
--------------
1) Naming scheme changed for auxillary jobs

Pegasus during the refinement of the abstract workflow to the
executable workflows adds auxillary jobs to do data stagein/stageout,
create work directories for workflow etc. The prefixes/suffixes added
for these jobs has been changed.

Additionally, the suffixes for the create dir jobs are now replaced
by prefixes

2) Staging of worker package to remote sites

Pegasus now supports staging of worker package as part of the workflow.

This feature is tracked through pegasus bugzilla .

http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=35

The worker package is staged automatically to the remote site, by
adding a setup transfer job to the workflow.

The setup transfer job by default uses GUC to stage the data.
However, this can be configured by setting the property
pegasus.transfer.setup.impl property. If you also, have
pegasus.transfer.*.impl set in your properties file, then you need
explicilty set pegasus.transfer.setup.impl to GUC

The code discovers the worker package by looking up pegasus::worker
in the transformation catalog.
Note: that the basename of the url's should not be changed. Pegasus
parses the basename to determine the version of the worker package.
Pegasus automatically determines the location of the worker package
to deploy on the remote site. Currently default mappings are as
follows
INTEL32 => x86
AMD64 => x86_64 or x86 if not available
INTEL64 =>x86

OS LINUX = rhel3

There is an untar job added to the workflow after the setup job that
un tars the worker package on the remote site. It defaults to /bin/tar .
However can be overriden by specifying the entry tar in the
transformation catalog for a particular site.

3) New Site Catalog Schema

This release of Pegasus has support for site catalog schema version 3.

HTML visualization of schema:
http://pegasus.isi.edu/mapper/docs/schemas/sc-3.0/sc-3.0.html

Schema itself:
http://pegasus.isi.edu/schema/sc-3.0.xsd

A sample xml file :
http://pegasus.isi.edu/schema/sc-3.0-sample.xml

To use a site catalog in the new format set
pegasus.catalog.site XML3

Changes to sc-client
sc-client command line tool was updated to convert an existing
site catalog from old format to the new format.

Sample usage
sc-client -i ldg-old-sites.xml -I XML -o ldg-new-sites.xml -O XML3
sc-client --help gives detailed help

4) pegasus-get-sites

pegasus-get-sites was recoded in JAVA and now generates the site
catalog confromant to schema sc-3.0.

Sample Usage to query VORS to generate a site catalog for OSG.
pegasus-get-sites --source VORS --grid osg -s ./sites-new.xml

The value passed to the source option is case sensitive.

Additionally, the VORS module of pegasus-get-sites determines the
value of GLOBUS_LOCATION variable dependant on whether the
auxillary jobmanager is of type fork or not.

If it is of type fork then picks up the value of GLOBUS_LOCATION
variable published in VORS for that site. else it picks up the
value from OSG_GRID variable published in VORS for that
site. i.e. GLOBUS_LOCATION is set to $OSG_GRID/globus

5) Overhaul of logging

The Pegasus logging interfaces have been reworked.
Now users can specify the logger they want to use, by specifying the
property pegasus.log.manager .

Currently, two logging implementations are supported.
Default - Pegasus homegrown logger that logs to stdout and stderr
directly.
Log4j - Uses log4j to log the messages.

The Log4j properties can be specified at runtime by specifying the
property pegasus.log.manager.log4j.conf

The format of the log message themselves can be specified at runtime
by specifying the property pegasus.log.manager.formatter

Right now two formatting modes are supported

a) Simple - This formats the messages in a simple format. The
messages are logged as is with minimal formatting. Below are
sample log messages in this format while ranking a dax according
to performance.

event.pegasus.ranking dax.id se18-gda.dax - STARTED
event.pegasus.parsing.dax dax.id se18-gda-nested.dax - STARTED
event.pegasus.parsing.dax dax.id se18-gda-nested.dax - FINISHED
job.id jobGDA
job.id jobGDA query.name getpredicted performace time 10.00
event.pegasus.ranking dax.id se18-gda.dax - FINISHED

b) Netlogger - This formats the messages in the Netlogger format ,
that is based on key value pairs. The netlogger format is useful
for loading the logs into a database to do some meaningful analysis.

Below are sample log messages in this format while ranking a dax
according to performance.

ts=2008-09-06T12:26:20.100502Z event=event.pegasus.ranking.start \
msgid=6bc49c1f-112e-4cdb-af54-3e0afb5d593c \
eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5 \
dax.id=se18-gda.dax prog=Pegasus

ts=2008-09-06T12:26:20.100750Z event=event.pegasus.parsing.dax.start \
msgid=fed3ebdf-68e6-4711-8224-a16bb1ad2969 \
eventId=event.pegasus.parsing.dax_887134a8-39cb-40f1-b11c-b49def0c5232\
dax.id=se18-gda-nested.dax prog=Pegasus

ts=2008-09-06T12:26:20.100894Z event=event.pegasus.parsing.dax.end \
msgid=a81e92ba-27df-451f-bb2b-b60d232ed1ad \
eventId=event.pegasus.parsing.dax_887134a8-39cb-40f1-b11c-b49def0c5232

ts=2008-09-06T12:26:20.100395Z event=event.pegasus.ranking \
msgid=4dcecb68-74fe-4fd5-aa9e-ea1cee88727d \
eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5 \
job.id="jobGDA"

ts=2008-09-06T12:26:20.100395Z event=event.pegasus.ranking \
msgid=4dcecb68-74fe-4fd5-aa9e-ea1cee88727d \
eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5 \
job.id="jobGDA" query.name="getpredicted performace" time="10.00"

ts=2008-09-06T12:26:20.102003Z event=event.pegasus.ranking.end \
msgid=31f50f39-efe2-47fc-9f4c-07121280cd64 \
eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5

6) New Transfer Refiner

Pegasus has a new transfer refiner named Cluster.

In this refinement strategy, clusters of stage-in and stageout jobs are
created per level of the workflow. It builds upon the Bundle refiner.

The differences between the Bundle and Cluster refiner are as follows.
- stagein is also clustered/bundled per level. In Bundle it was
for the whole workflow.
- keys that control the clustering ( old name bundling are )
cluster.stagein and cluster.stageout instead of bundle.stagein and
bundle.stageout

This refinement strategy also adds dependencies between the stagein
transfer jobs on different levels of the workflow to ensure that stagein
for the top level happens first and so on.

An image of the workflow with this refinement strategy can be found
at
http://vtcpc.isi.edu/pegasus/index.php/ChangeLog#Added_a_Cluster_Transfer_Refiner

7) New Transfer Implementation for GUC from globus 4.x

Pegasus has a new transfer implementation that allows it to use GUC
from globus 4.x series to transfer multiple files in one job.

In order to use this transfer implementation
- the property pegasus.transfer.*.impl must be set to value GUC.

There should be an entry in the transformation catalog with the
fully qualified name as globus::guc for all the sites where
workflow is run, or on the local site in case of third party
transfers.

Pegasus can automatically construct the path to the globus-url-copy
client, if the environment variable GLOBUS_LOCATION is specified in
the site catalog for the site.

The arguments with which the client is invoked can be specified
- by specifying the property pegasus.transfer.arguments
- associating the Pegasus profile key transfer.arguments

8) Recursive DAX'es

There is prototypical support for recursive dax'es. Recursive
DAX'es give you the ability to specify a job in the DAX that points
to another DAX that has to be executed.

There is a sample recursive dax at
$PEGASUS_HOME/examples/recursive.dax

The dax refers to pegasus jobs in turn plan and execute a dax

To get this dax planned by pegasus you will need to have additional
entries for dagman and pegasus in your transformation catalog.

For e.g.

local condor::dagman /opt/condor/7.1.0/bin/condor_dagman INSTALLED INTEL32::LINUX NULL
local pegasus::pegasus-plan:2.0 /lfs1/software/install/pegasus/default INSTALLED INTEL32::LINUX NULL

The recursive dax needs to be planned for site local, since the
pegasus itself runs on local site. The jobs in the dax specify -s
option where you want each of your workflows to run.

Recursive DAX do not need to contain only pegasus jobs. They can
contain application/normal jobs that one usually specifies in a
DAX. Pegasus determines that a particular job is planning and
execute job by looking for a pegasus profile key named type with
value recursive e.g.

<job id="ID0000003" namespace="pegasus" name="pegasus-plan" version="2.0">
<profile namespace="pegasus" key="type">recursive</profile>
<argument>-Dpegasus.user.properties=/lfs1/work/conf/properties
--dax /lfs1/work/dax3 -s tacc -o local --nocleanup --force
--rescue 1 --cluster horizontal -vvvvv --dir ./dag_3
</argument>
</job>

09) Rescue option to pegasus-plan for deferred planning

A rescue option to pegasus-plan has been added. The rescue option
takes in an integer value, that determines the number of times
rescue dags are submitted before re-planning is triggered in case
of failures in deferred planning. For this to work, Condor 7.1.0
or higher is required as it relies on the recently implemented
auto rescue feature in Condor DAGMan.

Even though re-planning is triggered, Condor DAGMan still ends up
submitting the rescue dag as it auto detects. The fix to it is to
remove the rescue dag files in case of re-planning. This is still
to be implemented

10) -j|--job-prefix option to pegasus-plan

pegasus-plan can now be passed the -j|--job-prefix option to
designate the prefix that needs to be used for constructing the
job submit file.

11) Executing workflows on Amazon EC2

Pegasus now has support of running workflows on EC2 with the
storage of files on S3. This feature is still in testing phase and
has not been tested fully.

To execute workflows on EC2/S3, Pegasus needs to be configured to
use S3 specific implementations of it's internal API's

a) First level Staging API - The S3 implementation stages in from the
local site ( submit node ) to a bucket on S3. Similarly the data is
staged back from the bucket to the local site ( submit node )
. All the first level transfers happen between the submit node
and the cloud. This means that input data can *only* be present
on the submit node when running on the cloud, and the output
data can be staged back only to the submit node.

b) Second Level Staging API - The S3 implementation retrieves input
data from the bucket to the worker node tmp directory and puts
created data back in the bucket.

c) Directory creation API - The S3 implementation creates a bucket
on S3 for the workflow instead of a directory.

d) Cleanup API - To cleanup files from the workflow specific bucket
on S3 during workflow execution.

The above implementations rely on s3cmd command line client to
interface with S3 filesystem. There should be an entry in the
transformation catalog with the fully qualified name as
amazon::s3cmd for the site corresponding to the cloud and the local
site.

To configure Pegasus to use these implementations set the following
properties

pegasus.transfer.*.impl S3
pegasus.transfer.sls.*.impl S3
pegasus.dir.create.impl S3
pegasus.file.cleanup.impl S3

pegasus.execute.*.filesystem.local true

12) Support for OSU Datacutter jobs

Pegasus has new gridstart mode called DCLauncher. This allows us to
launch the Data Cutter jobs using the wrapper that OSU group wrote.

Pegasus now supports the condor parallel universe.
To launch a job using DCLauncher, the following pegasus profile
keys need to be associated with the job
gridstart to DCLauncher
gridstart.path the path to the DCLauncher script

13) New Pegasus Profiles Keys
a) create.dir - this profile key triggers kicstart to create and
change directories before launching a job.

b) gridstart.path - this profile key specifies the path to the
gridstart used to launch a particular job

c) runtime - this profile key is useful when using Heft based site
selection. It allows users to associate expected runtimes of
jobs with the job description in DAX.

14) Kickstart captures machine information
Kickstart now logs machine information in the invocation record
that it creates for each job invocation. The Kickstart JAVA parser
can parse both records in old and new format.

A snippet of machine information captured is show below
<machine page-size="4096" provider="LINUX">
<stamp>2008-09-23T13:58:05.211-07:00</stamp>
<uname system="linux" nodename="viz-login" release="2.6.11.7"
machine="i686">#2 SMP Thu Apr 28 18:41:14 PDT 2005</uname>
<ram total="2125209600" free="591347712" shared="0" buffer="419291136"/>
<swap total="2006884352" free="2006876160"/>
<boot idle="943207.170">2008-09-12T12:03:49.772-07:00</boot>
<cpu count="4" speed="2400" vendor="GenuineIntel">Intel(R)
Xeon(TM) CPU 2.40GHz</cpu>
<load min1="0.07" min5="0.04" min15="0.00"/>
<proc total="110" running="1" sleeping="109" vmsize="614912000"
rss="206729216"/>
<task total="133" running="1" sleeping="132"/>
</machine>

15) Kickstart works in cygwin environment
Kickstart now compiles on cygwin. Kickstart could not find
SYS_NMLN variable in Cygwin to determine the uname datastructure's
size. Added a fix in the Makefile to add CFLAGS -DSYS_NMLN=20 when
the OS is Cygwin/Windows

The kickstart records generated on cygwin are slightly different
from the ones generated unix platforms. The kickstart parser was
modified to handle that.

The differences are as follows -
a) On cygwin inode value is double. The inode value is parsed as
double , but cast to long to prevent errors.

b) On cygwin the uid and gid values are long. They are passed as
long, but cast to int to prevent errors.

16) Changes to dirmanager
The dirmanager executable can now remove and create multiple
directories. This is achieved by specifying a whitespace separated
list of directories to the --dir option.

17) Added color-file option to showjob

There is now a --color-file option to show-job in
$PEGASUS_HOME/contrib/showlog to pass a file that has the mappings
from transformation name to colors.

The format of each line is as follows
transformation-name color

This can be used to assign different colors to compute jobs in a
workflow. The default color assigned is gray if none is
specified.

18) jobstate-summary tool

There is a new tool at $PEGASUS_HOME/bin/jobstate-summary.

It attempts to give a summary for the workflow. Should help in
jobstate-summ debugging failed job information. It will shows all
the information associated with a failed job. It gets the list of
failed job from the jobstate.log file. After that it parses latest
kickstart file for each failed job and show the exit code and all
the other information.

Usage: jobstate-summary --i <input directory> [--v(erbose)]
[--V(ersion)] [--h(elp)]

Input directory is the place where all the log files including jobstate.log file reside.
v option is for verbose debugging.
V option gives the pegasus version.
h option prints the help message.
A sample run is like jobstate-summary -i /dags/pegasus/diamond/run0013 -v

19) Support for DAGMan node categories

Pegasus now supports DAGMan node categories. DAGMan now allows to
specify CATEGORIES for jobs, and then specify tuning parameters (
like maxjobs ) per category. This functionality is exposed in
Pegasus as follows

The user can associate a dagman profile key category with the
jobs. The key attribute for the profile is category and value is
the category to which the job belongs to. For example you can set
the dagman category in the DAX for a job as follows

<job id="ID000001" namespace="vahi" name="preprocess" version="1.0" level="3" dv-namespace="vahi" dv-name="top" dv-version="1.0">
<profile namespace="dagman" key="CATEGORY">short-running</profile>
<argument>-a top -T 6 -i <filename file="david.f.a"/> -o
<filename file="vahi.f.b1"/>
<filename file="vahi.f.b2"/>
</argument>
<uses file="david.f.a" link="input" register="false"
transfer="true" type="data"/>
<uses file="vahi.f.b1" link="output" register="true"
transfer="true" />
<uses file="vahi.f.b2" link="output" register="true"
transfer="true" />
</job>

The property pegasus.dagman.[category].maxjobs can be used to
control the value.

For the above example, the user can set the property as follows

pegasus.dagman.short-running.maxjobs 2

In the DAG file generated you will see the category associated
with jobs. For the above example, it will look as follows

MAXJOBS short-running 2
CATEGORY preprocess_ID000001 short-running
JOB preprocess_ID000001 preprocess_ID000001.sub
RETRY preprocess_ID000001 2

20) Handling of pass through LFN

If a job in a DAX, specifies the same LFN as an input and an
output, it is a pass through LFN. Internally, the LFN is tagged
only as an input for the job. The reason for this, being that we
need to make sure that the replica catalog is queried for the
location of the LFN. If this is not handled specially, then LFN is
tagged internally as inout ( meaning it is generated during
workflow execution ). LFN's with type inout are not queried for in
the Replica Catalog in the force mode of operation

21) Tripping seqexec on first job failures

By default seqexec does not stop execution even if one of the
clustered jobs it is executing fails. This is because seqexec
tries to get as much work done as possible. If for some reason,
you want to make seqexec stop on first job failure, set the
following property in the properties file

pegasus.clusterer.job.aggregator.seqexec.firstjobfail true

22) New properties to choose the cleanup implementation

Two new properties were added to select the strategy and
implementation for file cleanup.

pegasus.file.cleanup.strategy
pegasus.file.cleanup.implementation

Currently there is only one cleanup strategy ( InPlace ) that can
be used and is loaded by default.

The cleanup implementations that can be used are
- Cleanup ( default)
- RM
- S3

Detailed documentation can be found at
$PEGASUS_HOME/etc/sample.properties.

23) New properties to choose the create dir implementation

The property pegasus.dir.create was deprecated.

It has been replaced by
pegasus.dir.create.strategy

Additionally, a user can specify a property to choose the
implementation used to create the directory on the remote sites.

pegasus.dir.create.impl

The create directory implementation that can be used are
- DefaultImplementation uses $PEGASUS_HOME/bin/dirmanager
executable to create a directory on the
remote site.
- S3 usese s3cmd to create a bucket on amazon S3.

BUGS FIXED
----------
1) Makefile for kickstart to build on Cygwin

Kickstart could not find SYS_NMLN variable in Cygwin to determine
the uname datastructure's size. Added a fix in the Makefile to add
CFLAGS -DSYS_NMLN=20 when the OS is Cygwin/Windows

2) Bug fix to getsystem release tools

Some systems have started using / in their system version name
which causes failures in Pegasus build process. Fixed the getsystem
release script which converts / into _

3) Bug fix in file cleanup module when stageout is enabled.

There was a bug in how the dependencies are added between the
stageout jobs and the file cleanup jobs. In certain cases, cleanup
could occur before the output was staged out. This is fixed now.

This bug was tracked through bugzilla
http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=37

4) Bug fix to deferred planning

Deferred planning used to fail if pegasus-plan was not given -o
option .

This is fixed now and was tracked through bugzilla
http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=34

5) Bug fix to caching on entries from Transformation Catalog

In certain cases, caching of entries did not work for the INSTALLED
case.

This is fixed now and was tracked through bugzilla
http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=33