Pegasus 2.2.0 Released

with No Comments
===============================
Release Notes for PEGASUS 2.2.0
===============================
 
NEW FEATURES
--------------
1) Naming scheme changed for auxillary jobs

   Pegasus during the refinement of the abstract workflow to the
   executable workflows adds auxillary jobs to do data stagein/stageout, 
   create work directories for workflow etc. The prefixes/suffixes added
   for these jobs has been changed.

   Type of Job			    |   Old Prefix	 | New Prefix
   -------------------------------------------------------------------
   Data Stage In Job		    |  rc_tx_		 | stage_in_
   Data Stage Out Job 	 	    |  new_rc_tx_	 | stage_out_
   Data Stage In Job between sites  |  inter_tx_	 | stage_inter_
   Data Registration Job	    |  new_rc_register_	 | register_
   Cleanup Job	     		    |  cln_	         | clean_up_   
   Transfer job to transfer the	    |  setup_tx_ 	 | stage_worker_
   worker package  	    	    |  			 |

   Additionally, the suffixes for the create dir jobs are now replaced 
   by prefixes

   Type of Job			    |   Old Suffix	 | New Prefix
   -------------------------------------------------------------------
   Directory creation job	    |  _cdir		 | create_dir_
   Synch Job in HourGlass mode	    |  pegasus_concat	 | pegasus_concat_
   

2) Staging of worker package to remote sites

   Pegasus now supports staging of worker package as part of the workflow.

   This feature is tracked through pegasus bugzilla .

   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=35

   The worker package is staged automatically to the remote site, by 
   adding a setup transfer job to the workflow. 

   The setup transfer job by default uses GUC to stage the data.
   However, this can be configured by setting the property 
   pegasus.transfer.setup.impl property. If you also, have 
   pegasus.transfer.*.impl set in your properties file, then you need 
   explicilty set pegasus.transfer.setup.impl to GUC


   The code discovers the worker package by looking up pegasus::worker
   in the transformation catalog. 
   Note: that the basename of the url's should not be changed. Pegasus
   parses the basename to determine the version of the worker package.
   Pegasus automatically determines the location of the worker package 
   to deploy on the remote site. Currently default mappings are as
   follows 
   INTEL32 => x86 
   AMD64 => x86_64 or x86 if not available 
   INTEL64 =>x86

   OS LINUX = rhel3

   There is an untar job added to the workflow after the setup job that
   un tars the worker package on the remote site. It defaults to /bin/tar .
   However can be overriden by specifying the entry tar in the
   transformation catalog for a particular site.


3) New Site Catalog Schema
   
   This release of Pegasus has support for site catalog schema version 3.

   HTML visualization of schema:
   http://pegasus.isi.edu/mapper/docs/schemas/sc-3.0/sc-3.0.html

   Schema itself:
   http://pegasus.isi.edu/schema/sc-3.0.xsd

   A sample xml file :
   http://pegasus.isi.edu/schema/sc-3.0-sample.xml

   To use a site catalog in the new format set
   pegasus.catalog.site  XML3   

   Changes to sc-client
   sc-client command line tool was updated to convert an existing
   site catalog from old format to the new format.

   Sample usage
   sc-client -i ldg-old-sites.xml -I XML -o ldg-new-sites.xml -O XML3
   sc-client --help gives detailed help

4) pegasus-get-sites
   
   pegasus-get-sites was recoded in JAVA and now generates the site 
   catalog confromant to schema sc-3.0. 

   Sample Usage to query VORS to generate a site catalog for OSG.
   pegasus-get-sites --source VORS --grid osg -s ./sites-new.xml

   The value passed to the source option is case sensitive. 
   
   Additionally, the VORS module of pegasus-get-sites determines the
   value of GLOBUS_LOCATION variable dependant on whether the
   auxillary jobmanager is of type fork or not. 

   If it is of type fork then picks up the value of GLOBUS_LOCATION
   variable published in VORS for that site. else it picks up the
   value from OSG_GRID variable published in VORS for that
   site. i.e. GLOBUS_LOCATION is set to $OSG_GRID/globus 

5) Overhaul of logging
  
   The Pegasus logging interfaces have been reworked. 
   Now users can specify the logger they want to use, by specifying the
   property pegasus.log.manager .

   Currently, two logging implementations are supported.
   Default - Pegasus homegrown logger that logs to stdout and stderr 
   	     directly.
   Log4j  -  Uses log4j to log the messages.
 
   The Log4j properties can be specified at runtime by specifying the
   property pegasus.log.manager.log4j.conf

   The format of the log message themselves can be specified at runtime
   by specifying the property pegasus.log.manager.formatter

   Right now two formatting modes are supported

   a) Simple - This formats the messages in a simple format. The 
      messages are logged as is with minimal formatting. Below are 
      sample log messages in this format while ranking a dax according 
      to performance.

      event.pegasus.ranking dax.id se18-gda.dax - STARTED
      event.pegasus.parsing.dax dax.id se18-gda-nested.dax - STARTED
      event.pegasus.parsing.dax dax.id se18-gda-nested.dax - FINISHED
      job.id jobGDA
      job.id jobGDA query.name getpredicted performace time 10.00
      event.pegasus.ranking dax.id se18-gda.dax - FINISHED

   b) Netlogger - This formats the messages in the Netlogger format , 
      that is based on key value pairs. The netlogger format is useful 
      for loading the logs into a database to do some meaningful analysis.

      Below are sample log messages in this format while ranking a dax 
      according to performance.

      ts=2008-09-06T12:26:20.100502Z event=event.pegasus.ranking.start \
      msgid=6bc49c1f-112e-4cdb-af54-3e0afb5d593c \
      eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5 \
      dax.id=se18-gda.dax prog=Pegasus

      ts=2008-09-06T12:26:20.100750Z event=event.pegasus.parsing.dax.start \
      msgid=fed3ebdf-68e6-4711-8224-a16bb1ad2969 \			   
      eventId=event.pegasus.parsing.dax_887134a8-39cb-40f1-b11c-b49def0c5232\
      dax.id=se18-gda-nested.dax prog=Pegasus
 
      ts=2008-09-06T12:26:20.100894Z event=event.pegasus.parsing.dax.end \
      msgid=a81e92ba-27df-451f-bb2b-b60d232ed1ad \
      eventId=event.pegasus.parsing.dax_887134a8-39cb-40f1-b11c-b49def0c5232

      ts=2008-09-06T12:26:20.100395Z event=event.pegasus.ranking \
      msgid=4dcecb68-74fe-4fd5-aa9e-ea1cee88727d \
      eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5 \
      job.id="jobGDA"

      ts=2008-09-06T12:26:20.100395Z event=event.pegasus.ranking \
      msgid=4dcecb68-74fe-4fd5-aa9e-ea1cee88727d \
      eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5 \
      job.id="jobGDA" query.name="getpredicted performace" time="10.00"

      ts=2008-09-06T12:26:20.102003Z event=event.pegasus.ranking.end \
      msgid=31f50f39-efe2-47fc-9f4c-07121280cd64 \
      eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5

6) New Transfer Refiner 

   Pegasus has a new transfer refiner named Cluster.

   In this refinement strategy, clusters of stage-in and stageout jobs are
   created per level of the workflow. It builds upon the Bundle refiner. 
   
   The differences between the Bundle and Cluster refiner are as follows.
   - stagein is also clustered/bundled per level. In Bundle it was
     for the whole workflow.
   - keys that control the clustering ( old name bundling are )
     cluster.stagein and cluster.stageout instead of bundle.stagein and
     bundle.stageout

   This refinement strategy also adds dependencies between the stagein 
   transfer jobs on different levels of the workflow to ensure that stagein
   for the top level happens first and so on.

   An image of the workflow with this refinement strategy can be found
   at 
   http://vtcpc.isi.edu/pegasus/index.php/ChangeLog#Added_a_Cluster_Transfer_Refiner

7) New Transfer Implementation for GUC from globus 4.x
   
   Pegasus has a new transfer implementation that allows it to use GUC
   from globus 4.x series to transfer multiple files in one job.
   
   In order to use this transfer implementation
      - the property pegasus.transfer.*.impl must be set to value GUC. 
  
   There should be an entry in the transformation catalog with the
   fully qualified  name as globus::guc for all the sites where
   workflow is run, or on the local site in case of third party
   transfers.
 
   Pegasus can automatically construct the path to the globus-url-copy
   client, if the environment variable GLOBUS_LOCATION is specified in
   the site catalog for the site. 

   The arguments with which the client is invoked can be specified
         - by specifying the property pegasus.transfer.arguments
         - associating the Pegasus profile key transfer.arguments
 

8) Recursive DAX'es
   
   There is prototypical support for recursive dax'es. Recursive
   DAX'es give you the ability to specify a job in the DAX that points
   to another DAX that has to be executed. 

   There is a sample recursive dax at
   $PEGASUS_HOME/examples/recursive.dax

   The dax refers to pegasus jobs in  turn plan and execute a dax

   To get this dax planned by pegasus you will need to have additional
   entries for dagman and pegasus in your transformation catalog.  

   For e.g. 

   local   condor::dagman  /opt/condor/7.1.0/bin/condor_dagman     INSTALLED       INTEL32::LINUX  NULL
   local  pegasus::pegasus-plan:2.0       /lfs1/software/install/pegasus/default  INSTALLED       INTEL32::LINUX  NULL

   The recursive dax needs to be planned for site local, since the
   pegasus itself runs on local site. The jobs in the dax specify -s
   option where you want each of your workflows to run.

   Recursive DAX do not need to contain only pegasus jobs. They can
   contain application/normal jobs that one usually specifies in a
   DAX. Pegasus determines that a particular job is planning and
   execute job by looking for a pegasus profile key named type with
   value recursive e.g.

   <job id="ID0000003" namespace="pegasus" name="pegasus-plan" version="2.0">
    <profile namespace="pegasus" key="type">recursive</profile>
    <argument>-Dpegasus.user.properties=/lfs1/work/conf/properties
    --dax /lfs1/work/dax3  -s tacc -o local --nocleanup  --force
    --rescue 1 --cluster horizontal -vvvvv --dir ./dag_3 
    </argument>
   </job>

09) Rescue option to pegasus-plan for deferred planning

    A rescue option to pegasus-plan has been added. The rescue option
    takes in an integer value, that determines the number of times
    rescue dags are submitted before re-planning is triggered in case
    of failures in deferred planning. For this to work, Condor 7.1.0
    or higher is required as it relies on the recently implemented
    auto rescue feature in Condor DAGMan. 

    Even though re-planning is triggered, Condor DAGMan still ends up
    submitting the rescue dag as it auto detects. The fix to it is to
    remove the  rescue dag files in case of re-planning. This is still
    to be implemented 

10) -j|--job-prefix option to pegasus-plan

    pegasus-plan can now be passed the -j|--job-prefix option to
    designate the prefix that needs to be used for constructing the
    job submit file.


11) Executing workflows on Amazon EC2
   
   Pegasus now has support of running workflows on EC2 with the
   storage of files on S3.  This feature is still in testing phase and
   has not been tested fully.

   To execute workflows on EC2/S3, Pegasus needs to be configured to
   use S3 specific implementations of it's internal API's

   a) First level Staging API - The S3 implementation stages in from the
      local site ( submit node ) to a bucket on S3. Similarly the data is
      staged back from the bucket to the local site ( submit node )
      . All the first level transfers happen between the submit node
      and the cloud. This means that input data can *only* be present
      on the submit node when running on the cloud, and the output
      data can be staged back only to the submit node.

   b) Second Level Staging API - The S3 implementation retrieves input
      data from the bucket to the worker node tmp directory and puts
      created data back in the bucket. 

   c) Directory creation API - The S3 implementation creates a bucket
      on S3 for the workflow instead of a directory. 

   d) Cleanup API - To cleanup files from the workflow specific bucket
      on S3 during workflow execution.
   
   The above implementations rely on s3cmd command line client to
   interface with S3 filesystem. There should be an entry in the
   transformation catalog with the fully qualified name as
   amazon::s3cmd for the site corresponding to the cloud and the local
   site. 

   To configure Pegasus to use these implementations set the following
   properties 

   pegasus.transfer.*.impl                      S3
   pegasus.transfer.sls.*.impl                S3
   pegasus.dir.create.impl                      S3
   pegasus.file.cleanup.impl                  S3

   pegasus.execute.*.filesystem.local   true


12) Support for OSU Datacutter jobs

   Pegasus has new gridstart mode called DCLauncher. This allows us to
   launch the Data Cutter jobs using the wrapper that OSU group wrote. 

   Pegasus now supports the condor parallel universe.
   To launch a job using DCLauncher, the following pegasus profile
   keys need to be associated with the job 
   gridstart          to DCLauncher
   gridstart.path     the path to the DCLauncher script

13) New Pegasus Profiles Keys
    a) create.dir - this profile key triggers kicstart to create and
       change directories before launching a job. 

    b) gridstart.path - this profile key specifies the path to the
       gridstart used to launch a particular job

    c) runtime - this profile key is useful when using Heft based site
       selection. It allows users to associate expected runtimes of
       jobs with the job description in DAX. 

14) Kickstart captures machine information 
    Kickstart now logs machine information in the invocation record
    that it creates for each job invocation. The Kickstart JAVA parser
    can parse both records in old and new format. 

    A snippet of machine information captured is show below
    <machine page-size="4096" provider="LINUX">
     <stamp>2008-09-23T13:58:05.211-07:00</stamp>
     <uname system="linux" nodename="viz-login" release="2.6.11.7"
       machine="i686">#2 SMP Thu Apr 28 18:41:14 PDT  2005</uname> 
     <ram total="2125209600" free="591347712" shared="0" buffer="419291136"/> 
     <swap total="2006884352" free="2006876160"/>
     <boot idle="943207.170">2008-09-12T12:03:49.772-07:00</boot>
     <cpu count="4" speed="2400" vendor="GenuineIntel">Intel(R)
       Xeon(TM) CPU 2.40GHz</cpu> 
     <load min1="0.07" min5="0.04" min15="0.00"/>
     <proc total="110" running="1" sleeping="109" vmsize="614912000"
      rss="206729216"/> 
     <task total="133" running="1" sleeping="132"/>
  </machine>

15) Kickstart works in cygwin environment
    Kickstart now compiles on cygwin. Kickstart could not find
    SYS_NMLN variable in Cygwin to determine the uname datastructure's
    size. Added a fix in the Makefile to add CFLAGS -DSYS_NMLN=20 when
    the OS is Cygwin/Windows 

    The kickstart records generated on cygwin are slightly different
    from the ones generated unix platforms. The kickstart parser was
    modified to handle that.
    
    The differences are as follows - 
    a) On cygwin inode value is double. The inode value is parsed as
    double , but cast to long to prevent errors. 

    b) On cygwin the uid and gid values are long. They are passed as
    long, but cast to int to prevent errors. 

16) Changes to dirmanager
    The dirmanager executable can now remove and create multiple
    directories. This is achieved by specifying a whitespace separated
    list of directories to the --dir option. 


17) Added color-file option to showjob

    There is now a  --color-file option to show-job in
    $PEGASUS_HOME/contrib/showlog to pass a file that has the mappings
    from transformation name to colors. 

    The format of each line is as follows
    transformation-name color

    This can be used to assign different colors to compute jobs in a
    workflow. The default color assigned is gray if none is
    specified. 

18) jobstate-summary tool

    There is a new tool at $PEGASUS_HOME/bin/jobstate-summary.

    It attempts to give a summary for the workflow. Should help in
    jobstate-summ debugging failed job information. It will shows all
    the information associated with a failed job. It gets the list of
    failed job from the jobstate.log file. After that it parses latest
    kickstart file for each failed job and show the exit code and all
    the other information. 

    Usage: jobstate-summary --i <input directory> [--v(erbose)]
           [--V(ersion)] [--h(elp)] 

    Input directory is the place where all the log files including jobstate.log file reside.
    v option is for verbose debugging.
    V option gives the pegasus version.
    h option prints the help message.
    A sample run is like jobstate-summary -i /dags/pegasus/diamond/run0013 -v


19) Support for DAGMan node categories


    Pegasus now supports DAGMan node categories. DAGMan now allows to
    specify CATEGORIES for jobs, and then specify tuning parameters (
    like maxjobs ) per category. This functionality is exposed in
    Pegasus as follows 

    The user can associate a dagman profile key category with the
    jobs. The key attribute for the profile is category and value is
    the category to which the job belongs to. For example you can set
    the dagman category in the DAX for a job as follows 

    <job id="ID000001" namespace="vahi" name="preprocess" version="1.0" level="3" dv-namespace="vahi" dv-name="top" dv-version="1.0">
       <profile namespace="dagman" key="CATEGORY">short-running</profile>
       <argument>-a top -T 6  -i <filename file="david.f.a"/>  -o
         <filename file="vahi.f.b1"/> 
         <filename file="vahi.f.b2"/>
       </argument>
       <uses file="david.f.a" link="input" register="false"
       transfer="true" type="data"/> 
       <uses file="vahi.f.b1" link="output" register="true"
       transfer="true" /> 
       <uses file="vahi.f.b2" link="output" register="true"
       transfer="true" /> 
    </job>

    The property pegasus.dagman.[category].maxjobs can be used to
    control the value. 

    For the above example, the user can set the property as follows 

    pegasus.dagman.short-running.maxjobs 2

    In the DAG file generated you will see the category associated
    with jobs. For the above example, it will look as follows 

    MAXJOBS short-running 2
    CATEGORY preprocess_ID000001 short-running
    JOB preprocess_ID000001 preprocess_ID000001.sub
    RETRY preprocess_ID000001 2

20) Handling of pass through LFN

    If a job in a DAX, specifies the same LFN as an input and an
    output, it is a pass through LFN. Internally, the LFN is tagged
    only as an input for the job. The reason for this, being that we
    need to make sure that the replica catalog is queried for the
    location of the LFN. If this is not handled specially, then LFN is
    tagged internally as inout ( meaning it is generated during
    workflow execution ). LFN's with type inout are not queried for in
    the Replica Catalog in the force mode of operation 


21) Tripping seqexec on first job failures

    By default seqexec does not stop execution even if one of the
    clustered jobs it is executing fails. This is because seqexec
    tries to get as much work done as possible. If for some reason,
    you want to make seqexec stop on first job failure, set the
    following property in the properties file 

    pegasus.clusterer.job.aggregator.seqexec.firstjobfail true


22) New properties to choose the cleanup implementation

    Two new properties were added to select the strategy and
    implementation for file cleanup.

    pegasus.file.cleanup.strategy
    pegasus.file.cleanup.implementation

    Currently there is only one cleanup strategy ( InPlace ) that can
    be used and is loaded by default. 

    The cleanup implementations that can be used are 
    	- Cleanup ( default)
	- RM
	- S3 

    Detailed documentation can be found at
    $PEGASUS_HOME/etc/sample.properties. 

23) New properties to choose the create dir implementation

    The property pegasus.dir.create was deprecated.

    It has been replaced by
    pegasus.dir.create.strategy   

    Additionally, a user can specify a property to choose the
    implementation used to create the directory on the remote sites. 

    pegasus.dir.create.impl   

    The create directory implementation that can be used are 
    - DefaultImplementation   uses $PEGASUS_HOME/bin/dirmanager
                              executable to create a directory on the
    			      remote site.  
    - S3   		      usese s3cmd to create a bucket on amazon S3.


BUGS FIXED
----------
1) Makefile for kickstart to build on Cygwin

   Kickstart could not find SYS_NMLN variable in Cygwin to determine
   the uname datastructure's size. Added a fix in the Makefile to add
   CFLAGS -DSYS_NMLN=20 when the OS is Cygwin/Windows 


2) Bug fix to getsystem release tools

   Some systems have started using / in their system version name
   which causes failures in Pegasus build process. Fixed the getsystem
   release script which converts / into _ 

3) Bug fix in file cleanup module when stageout is enabled.
   
   There was a bug in how the dependencies are added between the
   stageout jobs and the file cleanup jobs. In certain cases, cleanup
   could occur before the output was staged out. This is fixed now.

   This bug was tracked through bugzilla
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=37

4) Bug fix to deferred planning

   Deferred planning used to fail if pegasus-plan was not given -o
   option .

   This is fixed now and was tracked through bugzilla
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=34

5) Bug fix to caching on entries from Transformation Catalog

   In certain cases, caching of entries did not work for the INSTALLED
   case. 

   This is fixed now and was tracked through bugzilla
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=33