Download the corresponding disk image.

Launch Virtual Box on your machine. Follow the steps to add the vmdk file to Virtual Box and create a virtual machine inside the Virtual Box

We generate a 4 node diamond dax. There is a small piece of java code that uses the DAX API to generate the DAX. Open the file $HOME/pegasus-wms/dax/CreateDAX.java in a file editor:

$ vi dax/CreateDAX.java

There is a function Diamond( String site_handle, String pegasus_location ) that constructs the DAX. Towards the end of the function there is some commented out code.

 // Add analyze job
//To be uncommented for exercise 2.1
    
        Job j4 = new Job("j4", "pegasus", "analyze", "4.0");
        j4.addArgument("-a analyze -T 60 -i ").addArgument(fc1);
        j4.addArgument(" ").addArgument(fc2);
        j4.addArgument("-o ").addArgument(fd);
        j4.uses(fc1, File.LINK.INPUT);
        j4.uses(fc2, File.LINK.INPUT);
        j4.uses(fd, File.LINK.OUTPUT);
        
        //add job to the DAX
        dax.addJob(j4);

        //analyze job is a child to the findrange jobs
        dax.addDependency("j2", "j4");
        dax.addDependency("j3", "j4");
    
//End of commented out code for Exercise 2.1

The above snippet of code, adds a job with the ID0000004 to the DAX. It illustrates how to specify

After uncommenting the code, compile and run the CreateDAX program.

$ cd dax

$ javac -classpath .:/opt/pegasus/default/lib/pegasus.jar CreateDAX.java

$  java -classpath .:/opt/pegasus/default/lib/pegasus.jar CreateDAX local /opt/pegasus/default ./diamond.dax

Let us view the generated diamond.dax.

$ cat diamond.dax

Inside the DAX, you should see three sections.

First lets change to the tutorial base directory.

$ cd $HOME/pegasus-wms

In this exercise you will insert entries into the Replica Catalog. The replica catalog that we will use today is a simple file based catalog. We also support and recommend the following for production runs

A Replica Catalog maintains the LFN to PFN mapping for the input files of your workflow. Pegasus queries it to determine the locations of the raw input data files required by the workflow. Additionally, all the materialized data is registered into Replica Catalog for data reuse later on.

$ rm $HOME/pegasus-wms/config/rc.data

$  pegasus-rc-client  insert f.a \
          gsiftp://pegasus-vm/scratch/tutorial/inputdata/diamond/f.a pool=local

$ pegasus-rc-client  lookup f.a

 f.a gsiftp://pegasus-vm/scratch/tutorial/inputdata/diamond/f.a pool="local"

The site catalog contains information about the layout of your grid where you want to run your workflows. For each site following information is maintained

$ cat $HOME/pegasus-wms/config/sites.xml3

<sitecatalog xmlns="http://pegasus.isi.edu/schema/sitecatalog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:schemaLocation="http://pegasus.isi.edu/schema/sitecatalog http://pegasus.isi.edu/schema/sc-3.0.xsd" version="3.0">

<site  handle="cluster" arch="x86" os="LINUX" osrelease="" osversion="" glibc="">
  <grid  type="gt2" contact="pegasus-vm/jobmanager-fork" scheduler="Fork" jobtype="auxillary"/>
  <grid  type="gt2" contact="pegasus-vm/jobmanager-condor" scheduler="SGE" jobtype="compute"/>
  <head-fs>
   <scratch>
    <shared>
     <file-server protocol="gsiftp" url="gsiftp://pegasus" mount-point="/home/tutorial/cluster-scratch"/>
     <internal-mount-point mount-point="/home/tutorial/cluster-scratch"/>
    </shared>
   </scratch>
   <storage>
    <shared>
     <file-server protocol="gsiftp" url="gsiftp://pegasus" mount-point="/home/tutorial/cluster-storage"/>
     <internal-mount-point mount-point="/home/tutorial/cluster-storage"/>
    </shared>
   </storage>
  </head-fs>
  <replica-catalog  type="LRC" url="rlsn://localhost"/>
  <profile namespace="env" key="GLOBUS_LOCATION" >/opt/globus/default</profile>
  <profile namespace="env" key="JAVA_HOME" >/usr</profile>
  <profile namespace="env" key="LD_LIBRARY_PATH" >/opt/globus/default/lib</profile>
  <profile namespace="env" key="PEGASUS_HOME" >/opt/pegasus/default</profile>
  <profile namespace="pegasus" key="clusters.num" >1</profile>
  <profile namespace="pegasus" key="stagein.clusters" >1</profile>
 </site>

 <site  handle="local" arch="x86" os="LINUX" osrelease="" osversion="" glibc="">
  <grid  type="gt2" contact="localhost/jobmanager-fork" scheduler="Fork" jobtype="auxillary"/>
  <grid  type="gt2" contact="localhost/jobmanager-fork" scheduler="Fork" jobtype="compute"/>
  <head-fs>
   <scratch>
    <shared>
     <file-server protocol="gsiftp" url="file://" mount-point="/home/tutorial/local-scratch"/>
     <internal-mount-point mount-point="/home/tutorial/local-scratch"/>
    </shared>
   </scratch>
   <storage>
    <shared>
     <file-server protocol="gsiftp" url="file://" mount-point="/home/tutorial/local-storage"/>
     <internal-mount-point mount-point="/home/tutorial/local-storage"/>
    </shared>
   </storage>
  </head-fs>
  <replica-catalog  type="LRC" url="rlsn://localhost"/>
  <profile namespace="env" key="GLOBUS_LOCATION" >/opt/globus/default</profile>
  <profile namespace="env" key="JAVA_HOME" >/usr</profile>
  <profile namespace="env" key="LD_LIBRARY_PATH" >/opt/globus/default/lib</profile>
  <profile namespace="env" key="PEGASUS_HOME" >/opt/pegasus/default</profile>
 </site>

</sitecatalog>
 

$ [pegasus@pegasus pegasus-wms]$ pegasus-sc-client --vo engage --sc ./engage-osg-sc.xml \
  --source OSGMM --grid OSG -vvvv 

2010.11.24 18:00:46.410 PST: [INFO]  Skipping site CIT_CMS_T2 
2010.11.24 18:00:46.416 PST: [INFO]  Adding site RENCI-Engagement 
2010.11.24 18:00:46.475 PST: [INFO]  Adding site Nebraska 
2010.11.24 18:00:46.476 PST: [INFO]  Adding site Prairiefire 
2010.11.24 18:00:46.476 PST: [INFO]  Adding site BNL-ATLAS 
2010.11.24 18:00:46.477 PST: [INFO]  Adding site BNL-ATLAS__1 
2010.11.24 18:00:46.478 PST: [INFO]  Adding site UFlorida-PG 
2010.11.24 18:00:46.478 PST: [INFO]  Skipping site CIT_CMS_T2__1 
2010.11.24 18:00:46.478 PST: [INFO]  Adding site RENCI-Blueridge 
2010.11.24 18:00:46.479 PST: [INFO]  Adding site Nebraska__1 
2010.11.24 18:00:46.480 PST: [INFO]  Adding site UMissHEP 
2010.11.24 18:00:46.480 PST: [INFO]  Adding site UCR-HEP 
2010.11.24 18:00:46.481 PST: [INFO]  Adding site LIGO_UWM_NEMO 
2010.11.24 18:00:46.482 PST: [INFO]  Adding site FNAL_FERMIGRID 
2010.11.24 18:00:46.482 PST: [INFO]  Adding site USCMS-FNAL-WC1 
2010.11.24 18:00:46.483 PST: [INFO]  Adding site UConn-OSG 
2010.11.24 18:00:46.484 PST: [INFO]  Adding site UFlorida-HPC 
2010.11.24 18:00:46.484 PST: [INFO]  Adding site GridUNESP_CENTRAL 
2010.11.24 18:00:46.493 PST: [INFO]  Adding site NWICG_NotreDame 
2010.11.24 18:00:46.494 PST: [INFO]  Site LOCAL . Creating default entry 
2010.11.24 18:00:46.527 PST: [INFO]  Loaded 19 sites  
2010.11.24 18:00:46.527 PST:   Writing out site catalog to /home/tutorial/pegasus-wms/./engage-osg-sc.xml 
2010.11.24 18:00:46.959 PST:   Number of SRM Properties retrieved 14 
2010.11.24 18:00:46.970 PST:   Writing out properties to /home/tutorial/pegasus-wms/./pegasus.6475454308491531036.properties 
2010.11.24 18:00:46.972 PST: [INFO]  Time taken to execute is 1.101 seconds 
2010.11.24 18:00:46.972 PST: [INFO] event.pegasus.planner planner.version 3.0.0  - FINISHED 

The transformation catalog maintains information about where the application code resides on the grid. It also provides additional information about the transformation as to what system they are compiled for, what profiles or environment variables need to be set when the transformation is invoked etc.

$ cat $HOME/pegasus-wms/config/tc.data.text


# multiple line text-based transformation catalog: 2010-11-24T20:46:41.710-08:00
tr bin/mDiff {
        site local {
                profile env "MONTAGE_HOME" "." 
                pfn "gsiftp://pegasus-vm/scratch/tutorial/software/montage/3.0/x86/
bin/mDiff"
                arch "x86"
                os "LINUX"
                type "STAGEABLE"
        }
}

tr bin/mFitplane {
        site local {
                pfn "gsiftp://pegasus-vm/scratch/tutorial/software/montage/3.0/x86/
bin/mFitplane"
                arch "x86"
                os "LINUX"
                type "STAGEABLE"
        }
}

tr condor::dagman {
        site local {
                pfn "/usr/bin/condor_dagman"
                arch "x86"
                os "LINUX"
                type "INSTALLED"
        }
}

tr diamond::findrange:2.0 {
        site local {
                pfn "/opt/pegasus/default/bin/keg"
                arch "x86"
                os "LINUX"
                type "INSTALLED"
        }
}

tr diamond::preprocess:2.0 {
        site local {
                pfn "/opt/pegasus/default/bin/keg"
                arch "x86"
                os "LINUX"
                type "INSTALLED"
        }
}

tr mAdd:3.0 {
        site local {
                pfn "gsiftp://pegasus-vm/scratch/tutorial/software/montage/3.0/x86/
bin/mAdd"
                arch "x86"
                os "LINUX"
                type "STAGEABLE"
        }
}

tr mBackground:3.0 {
        site local {
                pfn "gsiftp://pegasus-vm/scratch/tutorial/software/montage/3.0/x86/
bin/mBackground"
                arch "x86"
                os "LINUX"
                type "STAGEABLE"
        }
}

tr mBgModel:3.0 {
        site local {
                pfn "gsiftp://pegasus-vm/scratch/tutorial/software/montage/3.0/x86/
bin/mBgModel"
                arch "x86"
                os "LINUX"
                type "STAGEABLE"
        }
}

tr mConcatFit:3.0 {
        site local {
                pfn "gsiftp://pegasus-vm/scratch/tutorial/software/montage/3.0/x86/
bin/mConcatFit"
                arch "x86"
                os "LINUX"
                type "STAGEABLE"
        }
}

tr mDiffFit:3.0 {
        site local {
                profile env "MONTAGE_HOME" "." 
                pfn "gsiftp://pegasus-vm/scratch/tutorial/software/montage/3.0/x86/
bin/mDiffFit"
                arch "x86"
                os "LINUX"
                type "STAGEABLE"
        }
}

tr mImgtbl:3.0 {
        site local {
                pfn "gsiftp://pegasus-vm/scratch/tutorial/software/montage/3.0/x86/
bin/mImgtbl"
                arch "x86"
                os "LINUX"
                type "STAGEABLE"
        }
}

tr mJPEG:3.0 {
        site local {
                pfn "gsiftp://pegasus-vm/scratch/tutorial/software/montage/3.0/x86/
bin/mJPEG"
                arch "x86"
                os "LINUX"
                type "STAGEABLE"
        }
}

tr mProjectPP:3.0 {
        site local {
                profile condor "priority" "25" 
                pfn "gsiftp://pegasus-vm/scratch/tutorial/software/montage/3.0/x86/
bin/mProjectPP"
                arch "x86"
                os "LINUX"
                type "STAGEABLE"
        }
}

tr mShrink:3.0 {
        site local {
                pfn "gsiftp://pegasus-vm/scratch/tutorial/software/montage/3.0/x86/
bin/mShrink"
                arch "x86"
                os "LINUX"
                type "STAGEABLE"
        }
}

Pegasus Workflow Planner is configured via the use of java properties. The instructors have provided a ready properties file at $HOME/.pegasusrc .

$ cat $HOME/.pegasusrc
 
##########################
# PEGASUS USER PROPERTIES 
##########################

## SELECT THE REPLICAT CATALOG MODE AND URL
pegasus.catalog.replica = File
pegasus.catalog.replica.file = ${user.home}/pegasus-wms/config/rc.data


## SELECT THE SITE CATALOG MODE AND FILE
pegasus.catalog.site = XML3
pegasus.catalog.site.file = ${user.home}/pegasus-wms/config/sites.xml3


## SELECT THE TRANSFORMATION CATALOG MODE AND FILE
pegasus.catalog.transformation = Text
pegasus.catalog.transformation.file = ${user.home}/pegasus-wms/config/tc.data.text

## USE DAGMAN RETRY FEATURE FOR FAILURES
dagman.retry=2

## CHECK JOB EXIT CODES FOR FAILURE
dagman.post.scope=all

## STAGE ALL OUR EXECUTABLES OR USE INSTALLED ONES 
pegasus.catalog.transformation.mapper = All

## WORK AND STORAGE DIR  
pegasus.dir.storage = storage
pegasus.dir.exec = exec

#JOB CATEGORIES
dagman.projection.maxjobs 2
[pegasus@pegasus pegasus-wms

In this exercise we are going to run pegasus-plan to generate a executable workflow from the abstract workflow (diamond.dax). The Executable workflow generated, are condor submit files that are submitted locally using pegasus-run

The instructors have provided:

You will need to write some things yourself, by following the instructions below:

Instructions:

We will change into the directory, that was mentioned by the output of pegasus-run command.

$ cd /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001

In this directory you will see a whole lot of files. That should not scare you. Unless things go wrong, you need to look at just a very few number of files to track the progress of the workflow

In this section, we will run the diamond workflow but remove the input file so that the workflow fails during execution. This is to highlight how to use pegasus-analyzer to debug a failed workflow.

First of all lets rename the input file f.a

 $ mv /scratch/tutorial/inputdata/diamond/f.a /scratch/tutorial/inputdata/diamond/f.a.old

 $ cd $HOME/pegasus-wms
 

We will now repeat exercise 2.4 and 2.5 and submit the workflow again.

Plan and Submit the diamond workflow . Pass --submit to pegasus-plan to submit in case of successful planning

$  pegasus-plan --dax `pwd`/dax/diamond.dax --force \
        --dir dags -s local -o local --nocleanup --submit -v


Use pegasus-status to track the workflow and wait it to fail

$ watch pegasus-status  /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002


-- Submitter: pegasus : <172.16.80.128:40195> : pegasus
 ID      OWNER/NODENAME   SUBMITTED     RUN_TIME ST PRI SIZE CMD               
  96.0   pegasus         7/19 17:40   0+00:01:06 R  0   7.3  condor_dagman -f -


The --long option to pegasus-status of a running workflow gives more detail
[pegasus@pegasus pegasus-wms]$ pegasus-status -l /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002
blackdiamond-0.dag is running.
11/25 01:25:06  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
11/25 01:25:06   ===     ===      ===     ===     ===        ===      ===
11/25 01:25:06     1       0        1       0       0          6        0

WORKFLOW STATUS : RUNNING | 1/8 ( 12% ) | (condor processing workflow)



We can also use --long option to pegasus-status to see the FINAL status of the workflow

$ pegasus-status -l /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002
blackdiamond-0.dag FAILED (status 1)
11/25 01:25:32  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
11/25 01:25:32   ===     ===      ===     ===     ===        ===      ===
11/25 01:25:32     1       0        0       0       0          6        1

WORKFLOW STATUS : FAILED | 1/8 ( 12% ) | (rescue needs to be submitted)

We will now run pegasus-analyzer on the failed workflow submit directory to see what job failed.

$ pegasus-analyzer  -i $HOME/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002
pegasus-analyzer: initializing...

************************************Summary*************************************

 Total jobs         :      8 (100.00%)
 # jobs succeeded   :      1 (12.50%)
 # jobs failed      :      1 (12.50%)
 # jobs unsubmitted :      6 (75.00%)

******************************Failed jobs' details******************************

=============================stage_in_local_local_0=============================

 last state: POST_SCRIPT_FAILURE
       site: local
submit file: /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002/stage_in_local_local_0.sub
output file: /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002/stage_in_local_local_0.out.002
 error file: /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002/stage_in_local_local_0.err.002

**************************************Done**************************************

pegasus-analyzer: end of status report

[pegasus@pegasus pegasus-wms]$ pegasus-analyzer  -i /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002
pegasus-analyzer: initializing...

************************************Summary*************************************

 Total jobs         :      8 (100.00%)
 # jobs succeeded   :      1 (12.50%)
 # jobs failed      :      1 (12.50%)
 # jobs unsubmitted :      6 (75.00%)

******************************Failed jobs' details******************************

=============================stage_in_local_local_0=============================

 last state: POST_SCRIPT_FAILURE
       site: local
submit file: /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002/stage_in_local_local_0.sub
output file: /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002/stage_in_local_local_0.out.002
 error file: /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002/stage_in_local_local_0.err.002

-------------------------------Task #1 - Summary--------------------------------

site        : local
hostname    : pegasus
executable  : /opt/pegasus/default/bin/pegasus-transfer
arguments   : 
exitcode    : 1
working dir : /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002

--Task #1 - pegasus::pegasus-transfer - pegasus::pegasus-transfer:1.0 - stdout--

2010-11-25 01:25:22,320    INFO:  Reading URL pairs from stdin
2010-11-25 01:25:22,321    INFO:  PATH=/usr/local/globus/default/bin:/opt/pegasus/default/bin:/usr/bin:/bin
2010-11-25 01:25:22,321    INFO:  LD_LIBRARY_PATH=/usr/local/globus/default/lib:/usr/java/jdk1.6.0_20/jre/lib/amd64
2010-11-25 01:25:22,321    INFO:  Executing cp commands
/bin/cp: cannot stat `/scratch/tutorial/inputdata/diamond/f.a': No such file or directory
2010-11-25 01:25:22,331 CRITICAL:  Command'/bin/cp -L"/scratch/tutorial/inputdata/diamond/f.a"
                 "/home/tutorial/local-scratch/exec/pegasus/pegasus/blackdiamond/run0002/f.a"'failed with error code 1

**************************************Done**************************************

pegasus-analyzer: end of status report

[pegasus@pegasus pegasus-wms]$ 

The above tells us that the stage-in job for the workflow failed, and points us to the stdout of the job. By default, all jobs in Pegasus are launched via kickstart that captures runtime provenance of the job and helps in debugging. Hence, the stdout of the job is the kickstart stdout which is in XML.

. the duration of the job the start time for the job the node on which the job ran the stdout/stderr of the job the arguments with which it launched the job the environment that was set for the job before it was launched. the machine information about the node that the job ran on Amongst the above information, the dagman.out file gives a coarser grained estimate of the job duration and start time

This section explains how to read kickstart output and DAGMan Condor log files.

$ cat /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002/stage_in_local_local_0.out.002 

 <?xml version="1.0" encoding="ISO-8859-1"?>
 <invocation xmlns="http://pegasus.isi.edu/schema/invocation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://pegasus.isi.edu/schema/invocation http://pegasus.isi.edu/schema/iv-2.1.xsd" version="2.1"
  start="2010-11-29T19:10:23.862-08:00" duration="0.076" transformation="pegasus::pegasus-transfer" 
  derivation="pegasus::pegasus-transfer:1.0" resource="local" wf-label="blackdiamond" 
  wf-stamp="2010-11-29T18:57:59-08:00" interface="eth0" hostaddr="10.0.2.15" hostname="pegasus-vm.local" 
  pid="5428" uid="501" user="pegasus" gid="501" group="pegasus" umask="0022">
 
 <mainjob start="2010-11-29T19:10:23.876-08:00" duration="0.063" pid="5429">
    <usage utime="0.040" stime="0.023" minflt="2758" majflt="0" nswap="0" nsignals="0" nvcsw="5" nivcsw="20"/>
    <status raw="256"><regular exitcode="1"/></status>
    <statcall error="0">
      <file name="/opt/pegasus/default/bin/pegasus-transfer">23212F7573722F62696E2F656E762070</file>
      <statinfo mode="0100775" size="25314" inode="2022205" nlink="1" blksize="4096" blocks="64" 
               mtime="2010-11-23T13:14:52-08:00" 
             atime="2010-11-29T19:10:07-08:00" ctime="2010-11-25T00:01:52-08:00" uid="501" user="pegasus" 
               gid="501" group="pegasus"/>
    </statcall>
    <argument-vector/>
  </mainjob>
  <cwd>/home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0002</cwd>
  <usage utime="0.002" stime="0.013" minflt="475" majflt="0" nswap="0" nsignals="0" nvcsw="1" nivcsw="5"/>
  

   <machine page-size="4096">
    <stamp>2010-12-23T10:56:43.817-08:00</stamp>
    <uname system="linux" nodename="pegasus-vm" release="2.6.32-5-686" machine="i686">
      #1 SMP Fri Dec 10 16:12:40 UTC 2010</uname>
   <linux>
    <ram total="527044608" free="242290688" shared="0" buffer="41041920"/>
    <swap total="417325056" free="417325056"/>
    <boot idle="1597.500">2010-12-23T10:29:16.599-08:00</boot>
    <cpu count="1" speed="2797" vendor="GenuineIntel">Intel(R) Xeon(R) CPU E5462 @ 2.80GHz</cpu>
    <load min1="0.05" min5="0.02" min15="0.00"/>
    <proc total="88" running="1" sleeping="87" vmsize="344793088" rss="123768832"/>
    <task total="101" running="1" sleeping="100"/>
   </linux>
  </machine>
 

  <statcall error="0" id="stdin">
    <descriptor number="0"/>
    <statinfo mode="0100664" size="142" inode="2250032" nlink="1" blksize="4096" blocks="16" 
     mtime="2010-11-29T19:09:20-08:00"   atime="2010-11-29T19:10:07-08:00" ctime="2010-11-29T19:09:20-08:00" 
     uid="501" user="pegasus" gid="501" group="pegasus"/>
  </statcall>

  <statcall error="0" id="stdout">
    <temporary name="/opt/condor/local.pegasus/spool/local_univ_execute/dir_5427/gs.out.awOX6p" descriptor="3"/>
    <statinfo mode="0100600" size="762" inode="2054511" nlink="1" blksize="4096" blocks="16" 
          mtime="2010-11-29T19:10:23-08:00" atime="2010-11-29T19:10:23-08:00" ctime="2010-11-29T19:10:23-08:00" 
             uid="501" user="pegasus" gid="501" group="pegasus"/>
    <data>2010-11-29 19:10:23,920    INFO:  Reading URL pairs from stdin
2010-11-29 19:10:23,921    INFO:  PATH=/usr/local/globus/default/bin:/opt/pegasus/default/bin:/usr/bin:/bin
2010-11-29 19:10:23,921    INFO:  LD_LIBRARY_PATH=/usr/local/globus/default/lib:/usr/java/jdk1.6.0_20/jre/lib/amd64/
2010-11-29 19:10:23,921    INFO:  Executing cp commands
/bin/cp: cannot stat `/scratch/tutorial/inputdata/diamond/f.a&apos;: No such file or directory
2010-11-29 19:10:23,932 CRITICAL:  Command &apos;/bin/cp -L &quot;/scratch/tutorial/inputdata/diamond/f.a&quot; 
    &quot;/home/tutorial/local-scratch/exec/pegasus/pegasus/blackdiamond/run0002/f.a&quot;&apos; failed with error code 1
</data>
  </statcall>

  <statcall error="0" id="stderr">
    <temporary name="/opt/condor/local.pegasus/spool/local_univ_execute/dir_5427/gs.err.oz9MOG" descriptor="4"/>
    <statinfo mode="0100600" size="0" inode="2054512" nlink="1" blksize="4096" blocks="8" 
    mtime="2010-11-29T19:10:23-08:00"  atime="2010-11-29T19:10:23-08:00" ctime="2010-11-29T19:10:23-08:00" 
    uid="501" user="pegasus" gid="501" group="pegasus"/>
  </statcall>

  <statcall error="2" id="gridstart">
    <!-- ignore above error -->
    <file name="condor_exec.exe"/>
  </statcall>
  <statcall error="0" id="logfile">
    <descriptor number="1"/>
    <statinfo mode="0100644" size="0" inode="2250072" nlink="1" blksize="4096" blocks="8" mtime="2010-11-29T19:10:23-08:00" 
    atime="2010-11-29T19:10:23-08:00" ctime="2010-11-29T19:10:23-08:00" uid="501" user="pegasus" gid="501" group="pegasus"/>
  </statcall>
  <statcall error="0" id="channel">
    <fifo name="/opt/condor/local.pegasus/spool/local_univ_execute/dir_5427/gs.app.qCOCwX" descriptor="5" count="0"
     rsize="0" wsize="0"/>
    <statinfo mode="010640" size="0" inode="2054524" nlink="1" blksize="4096" blocks="8" mtime="2010-11-29T19:10:23-08:00" 
     atime="2010-11-29T19:10:23-08:00" ctime="2010-11-29T19:10:23-08:00" uid="501" user="pegasus" gid="501" 
    group="pegasus"/>
  </statcall>
  <environment>
    <env key="GLOBUS_LOCATION">/usr/local/globus/default</env>
    <env key="GRIDSTART_CHANNEL">/opt/condor/local.pegasus/spool/local_univ_execute/dir_5427/gs.app.qCOCwX</env>
    <env key="JAVA_HOME">/usr</env>
    <env key="LD_LIBRARY_PATH">/usr/java/jdk1.6.0_20/jre/lib/amd64/server:/usr/java/jdk1.6.0_20/jre/lib/amd64:</env>
    <env key="PEGASUS_HOME">/opt/pegasus/default</env>
    <env key="TEMP">/opt/condor/local.pegasus/spool/local_univ_execute/dir_5427</env>
    <env key="TMP">/opt/condor/local.pegasus/spool/local_univ_execute/dir_5427</env>
    <env key="TMPDIR">/opt/condor/local.pegasus/spool/local_univ_execute/dir_5427</env>
    <env key="_CONDOR_ANCESTOR_4843">4862:1291085504:2790807554</env>
    <env key="_CONDOR_ANCESTOR_4862">5427:1291086623:1798288782</env>
    <env key="_CONDOR_ANCESTOR_5427">5428:1291086623:2750667008</env>
    <env key="_CONDOR_HIGHPORT">41000</env>
    <env key="_CONDOR_JOB_AD">/opt/condor/local.pegasus/spool/local_univ_execute/dir_5427/.job.ad</env>
    <env key="_CONDOR_LOWPORT">40000</env>
    <env key="_CONDOR_MACHINE_AD">/opt/condor/local.pegasus/spool/local_univ_execute/dir_5427/.machine.ad</env>
    <env key="_CONDOR_SCRATCH_DIR">/opt/condor/local.pegasus/spool/local_univ_execute/dir_5427</env>
    <env key="_CONDOR_SLOT">1</env>
  </environment>
  <resource>
    <soft id="RLIMIT_CPU">unlimited</soft>
    <hard id="RLIMIT_CPU">unlimited</hard>
    <soft id="RLIMIT_FSIZE">unlimited</soft>
    <hard id="RLIMIT_FSIZE">unlimited</hard>
    <soft id="RLIMIT_DATA">unlimited</soft>
    <hard id="RLIMIT_DATA">unlimited</hard>
    <soft id="RLIMIT_STACK">unlimited</soft>
    <hard id="RLIMIT_STACK">unlimited</hard>
    <soft id="RLIMIT_CORE">0</soft>
    <hard id="RLIMIT_CORE">0</hard>
    <soft id="RESOURCE_5">unlimited</soft>
    <hard id="RESOURCE_5">unlimited</hard>
    <soft id="RLIMIT_NPROC">unlimited</soft>
    <hard id="RLIMIT_NPROC">unlimited</hard>
    <soft id="RLIMIT_NOFILE">1024</soft>
    <hard id="RLIMIT_NOFILE">1024</hard>
    <soft id="RLIMIT_MEMLOCK">32768</soft>
    <hard id="RLIMIT_MEMLOCK">32768</hard>
    <soft id="RLIMIT_AS">unlimited</soft>
    <hard id="RLIMIT_AS">unlimited</hard>
    <soft id="RLIMIT_LOCKS">unlimited</soft>
    <hard id="RLIMIT_LOCKS">unlimited</hard>
    <soft id="RLIMIT_SIGPENDING">8192</soft>
    <hard id="RLIMIT_SIGPENDING">8192</hard>
    <soft id="RLIMIT_MSGQUEUE">819200</soft>
    <hard id="RLIMIT_MSGQUEUE">819200</hard>
    <soft id="RLIMIT_NICE">0</soft>
    <hard id="RLIMIT_NICE">0</hard>
    <soft id="RLIMIT_RTPRIO">0</soft>
    <hard id="RLIMIT_RTPRIO">0</hard>
  </resource>
</invocation>

Sometimes you may want to halt the execution of the workflow or just permanently remove it. You can stop/halt a workflow by running the pegasus-remove command mentioned in the output of pegasus-run

$ pegasus-remove $HOME/pegasus-wms/dags/tutorial/pegasus/diamond/runXXXX

Job 2788.0 marked for removal

In this section, we will generate statistics and plots of the diamond workflow we ran using pegasus-statistics and pegasus-plots

$ cd $HOME/pegasus-wms


$ pegasus-statistics dags/tutorial/pegasus/blackdiamond/run0001/


tutorial@pegasus-vm:~/pegasus-wms$ pegasus-statistics dags/tutorial/pegasus/blackdiamond/run0001/



******************************************** SUMMARY ********************************************
#Legends
#Workflow runtime (min,sec) - the waltime from the start of the workflow execution to the end as 
                              reported by the DAGMAN.In case of rescue dag the value is the cumulative 
                              of all retries.
#Cumulative workflow runtime (min,sec) - the sum of the walltime of all jobs as reported by the DAGMan .
                                         In case of job retries the value is the cumulative of all retries.

Job summary
  Total - the total number of jobs in the workflow. The total number of jobs is calculated by parsing 
          the .dag file.  For workflows having SUBDAX jobs , the SUDBAX job is skipped , but the 
          calculation takes into account all the jobs that make up the SUBDAX sub workflow. 
          For workflows having SUBDAG jobs , the SUBDAG jobs are treated like regular jobs.
  Succeeded - the total number of succeeded jobs in the workflow . 
  Failed - the total number of failed jobs in the workflow .
  Unsubmitted - the total number of unsubmitted jobs in the workflow .
  Unknown - the total number of jobs that are submitted, but has not completed execution or the state
            is unknown in the workflow.

SUBDAX summary
  Total - the total number of SUBDAX jobs in the workflow 
  Succeeded - the total number of succeeded SUBDAX jobs in the workflow.
  Failed - the total number of failed SUBDAX jobs in the workflow.
  Unsubmitted - the total number of unsubmitted SUBDAX jobs in the workflow.
  Unknown - the total number of SUBDAX jobs that are submitted, but has not completed execution or
            the state is unknown in the workflow.

Workflow runtime                   :            5 min.  5 sec.
Cumulative workflow runtime        :            4 min.  0 sec.

          Total        Succeeded    Failed       Unsubmitted  Unknown        
Jobs      8            8            0            0            0              
SUBDAX    0            0            0            0            0              

Workflow execution statistics :
/home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/statistics/workflow.txt

Job statistics : 
/home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/statistics/jobs.txt

Workflow events with time starting from zero : 
/home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/statistics/jobstate.txt

Logical transformation statistics :
/home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/statistics/breakdown.txt
**************************************************************************************************

 

1.3.5.2. Generating plots using pegasus-plots

pegasus-plots generates graphs and charts to visualize workflow execution. To generate graphs and charts run the command as shown below.

$ cd $HOME/pegasus-wms 
$ pegasus-plots dags/tutorial/pegasus/blackdiamond/run0001/

******  show-job *****  
Please wait, this may take a few minutes ...
****** Finished executing show-job  ***** 
******  show-run *****  
Please wait, this may take a few minutes ...
****** Finished executing show-run  ***** 
******  dag2dot  ***** 
Please wait, this may take a few minutes ...
****** Finished executing dag2dot ***** 
******  dot  ***** 
****** Finished executing dot2png ***** 
******  dax2dot  ***** 
Please wait, this may take a few minutes ...
****** Finished executing dax2dot ***** 
******  dot  ***** 
****** Finished executing dot2png ***** 



******************************************** SUMMARY ********************************************

DAX graph - 
png format : /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/graph/diamond-dax.png 
eps format : /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/graph/diamond-dax.eps  

DAG graph - 
png format : /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/graph/blackdiamond-dag.png 
eps format : /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/graph/blackdiamond-dag.eps 

Workflow execution Gantt chart -
png format : Failed to generate png format.Application 'convert' not available.
eps format : /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/graph/blackdiamond-2.eps

Host over time chart -
png format : Failed to generate png format.Application 'convert' not available.
eps format : /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/graph/blackdiamond-host.eps 
**************************************************************************************************


[pegasus@pegasus pegasus-wms]$ 

1.3.5.2.1. Abstract Worfklow / DAX Image

/home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/graph/diamond-dax.png

Figure 1.1. Figure: Black Diamond DAX Image

1.3.5.2.2. Executable Workflow / DAG Image

/home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/graph/blackdiamond-dag.png

Figure 1.2. Figure: Black Diamond DAG Image

1.3.5.2.3. Gantt Chart of Workflow Execution

/home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/graph/blackdiamond-2.png

X axis - time in seconds . Each tic is 60 seconds

Y axis - Job Number .

Figure 1.3. Figure: Gantt Chart of Workflow Execution

1.3.5.2.4. Host over time chart

/home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0001/graph/blackdiamond-host.png

X axis - time in seconds . Each tic is 60 seconds

Y axis - Job Number .

Figure 1.4. Figure: Gantt Chart of Workflow Execution

Sometimes a workflow may have too many jobs whose execution time is a few seconds long. In such instances the overhead of scheduling each job on a grid is too large and the runtime of the entire workflow can be optimized by using Pegasus clustering techniques. One such technique is to cluster jobs horizontally on the same level into one or more sequential jobs.

$ cd $HOME/pegasus-wms
$ pegasus-plan -Dpegasus.schema.dax=/opt/pegasus/default/etc/dax-2.1.xsd \
            --dir `pwd`/dags --sites cluster --output local --nocleanup --force\
            --cluster horizontal --dax `pwd`/dax/montage.dax -v

After clustering the executable workflow will contain 26 jobs compared to 44 in the non clustered mode.

In the DAX you can specify what output data products you want to track in the replica catalog. This is done by setting the register flags with the output files for a job. For our tutorial, we only register the final output data products. So if you were able to execute the diamond or the montage workflow successfully, we can do data reuse. Let us run pegasus-plan on the diamond workflow again. However, this time we will remove the --force option.

$ cd $HOME/pegasus-wms
$ pegasus-plan --dax `pwd`/dax/diamond.dax --dir `pwd`/dags -s local -o local --nocleanup -v

2010.11.25 01:35:11.186 PST: [INFO] event.pegasus.refinement dax.id blackdiamond_0  - STARTED 
2010.11.25 01:35:11.210 PST: [INFO] event.pegasus.reduce dax.id blackdiamond_0  - STARTED 
2010.11.25 01:35:11.211 PST: [INFO]  Nodes/Jobs Deleted from the Workflow during reduction  
2010.11.25 01:35:11.211 PST: [INFO]     analyze_j4 
2010.11.25 01:35:11.211 PST: [INFO]     findrange_j2 
2010.11.25 01:35:11.211 PST: [INFO]     findrange_j3 
2010.11.25 01:35:11.211 PST: [INFO]     preprocess_j1 
2010.11.25 01:35:11.211 PST: [INFO]  Nodes/Jobs Deleted from the Workflow during reduction  - DONE 
2010.11.25 01:35:11.212 PST: [INFO] event.pegasus.reduce dax.id blackdiamond_0  - FINISHED 
2010.11.25 01:35:11.212 PST: [INFO] event.pegasus.siteselection dax.id blackdiamond_0  - STARTED 
2010.11.25 01:35:11.219 PST: [INFO] event.pegasus.siteselection dax.id blackdiamond_0  - FINISHED 
2010.11.25 01:35:11.289 PST: [INFO]  Grafting transfer nodes in the workflow 
2010.11.25 01:35:11.290 PST: [INFO] event.pegasus.generate.transfer-nodes dax.id blackdiamond_0  - STARTED 
2010.11.25 01:35:11.370 PST: [INFO]  Adding stage out jobs for jobs deleted from the workflow 
2010.11.25 01:35:11.370 PST: [INFO]  The leaf file f.d is already at the output pool local 
2010.11.25 01:35:11.371 PST: [INFO] event.pegasus.generate.transfer-nodes dax.id blackdiamond_0  - FINISHED 
2010.11.25 01:35:11.372 PST: [INFO] event.pegasus.generate.workdir-nodes dax.id blackdiamond_0  - STARTED 
2010.11.25 01:35:11.374 PST: [INFO] event.pegasus.generate.workdir-nodes dax.id blackdiamond_0  - FINISHED 
2010.11.25 01:35:11.374 PST: [INFO] event.pegasus.generate.cleanup-wf dax.id blackdiamond_0  - STARTED 
2010.11.25 01:35:11.375 PST: [INFO] event.pegasus.generate.cleanup-wf dax.id blackdiamond_0  - FINISHED 
2010.11.25 01:35:11.375 PST: [INFO] event.pegasus.refinement dax.id blackdiamond_0  - FINISHED 
2010.11.25 01:35:11.426 PST: [INFO]  Generating codes for the concrete workflow 
2010.11.25 01:35:12.078 PST: [INFO]  Generating codes for the concrete workflow -DONE 
2010.11.25 01:35:12.083 PST:   


The executable workflow generated contains only a single NOOP job.
It seems that the output files are already at the output site. 
To regenerate the output data from scratch specify --force option.



pegasus-run -Dpegasus.user.properties=$HOME/.../blackdiamond/run0003/pegasus.4078026914028890643.properties\
 /home/tutorial/pegasus-wms/dags/tutorial/pegasus/blackdiamond/run0003

 
2010.11.25 01:35:12.083 PST:   Time taken to execute is 1.508 seconds 
2010.11.25 01:35:12.083 PST: [INFO] event.pegasus.planner planner.version 3.0.0  - FINISHED 

You can increase the debug level to see how pegasus deletes the jobs bottom up of the workflow. Pass -vvvv to pegasus-plan command.

Pegasus 3.0 allows you to create workflows of workflows i.e your workflow can contain dax jobs that refer to the sub-workflows. In this exercise, we will execute a workflow super-diamond that will execute two diamond workflows.

Let us look at superdiamond.dax in the dax directory

$ cat $HOME/pegasus-wms/dax/superdiamond.dax

<?xml version="1.0" encoding="UTF-8"?>
<!-- generated on: 2010-11-25T08:42:30-08:00 -->
<!-- generated by: pegasus [ ?? ] -->
<adag xmlns="http://pegasus.isi.edu/schema/DAX" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://pegasus.isi.edu/schema/DAX http://pegasus.isi.edu/schema/dax-3.2.xsd" versi
on="3.2" name="superdiamond" index="0" count="1">

<!-- Section 1: Files - Acts as a Replica Catalog (can be empty) -->

   <file name="f.a">
      <pfn url="file:///scratch/tutorial/inputdata/diamond/f.a" site="local"/>
   </file>
   
   <file name="black-1.dax">
      <pfn url="/home/tutorial/pegasus-wms/dax/black-1.dax" site="local"/>
   </file>

   <file name="black-2.dax">
      <pfn url="/home/tutorial/pegasus-wms/dax/black-2.dax" site="local"/>
   </file>


<!-- Section 2: Executables - Acts as a Transformaton Catalog (can be empty) -->


<!-- Section 3: Transformations - Aggregates executables and Files (can be empty) -->


<!-- Section 4: Job's, DAX's or Dag's - Defines a JOB or DAX or DAG (Atleast 1 required) -->

   <dax id="d1" file="black-1.dax"  >
    <argument>-s local --force -o local</argument>
   </dax>

   <dax id="d2" file="black-2.dax"  >
    <argument>-s local --force -o local</argument>
   </dax>



<!-- Section 5: Dependencies - Parent Child relationships (can be empty) -->

   <child ref="d2">
      <parent ref="d1"/>
   </child>

</adag>

Now let us submit this super diamond workflow

$ pegasus-plan --dax `pwd`/dax/superdiamond.dax --force --submit\
               --dir dags -s local -o local --nocleanup -v

2010.11.29 21:15:49.110 PST: [INFO] event.pegasus.refinement dax.id superdiamond_0  - STARTED 
2010.11.29 21:15:49.123 PST: [INFO] event.pegasus.siteselection dax.id superdiamond_0  - STARTED 
2010.11.29 21:15:49.142 PST: [INFO] event.pegasus.siteselection dax.id superdiamond_0  - FINISHED 
2010.11.29 21:15:49.220 PST: [INFO]  Grafting transfer nodes in the workflow 
2010.11.29 21:15:49.221 PST: [INFO] event.pegasus.generate.transfer-nodes dax.id superdiamond_0  - STARTED 
2010.11.29 21:15:49.305 PST: [INFO] event.pegasus.generate.transfer-nodes dax.id superdiamond_0  - FINISHED 
2010.11.29 21:15:49.307 PST: [INFO] event.pegasus.generate.workdir-nodes dax.id superdiamond_0  - STARTED 
2010.11.29 21:15:49.312 PST: [INFO] event.pegasus.generate.workdir-nodes dax.id superdiamond_0  - FINISHED 
2010.11.29 21:15:49.312 PST: [INFO] event.pegasus.generate.cleanup-wf dax.id superdiamond_0  - STARTED 
2010.11.29 21:15:49.314 PST: [INFO] event.pegasus.generate.cleanup-wf dax.id superdiamond_0  - FINISHED 
2010.11.29 21:15:49.314 PST: [INFO] event.pegasus.refinement dax.id superdiamond_0  - FINISHED 
2010.11.29 21:15:49.371 PST: [INFO]  Generating codes for the concrete workflow 
2010.11.29 21:15:50.200 PST: [INFO]  Generating codes for the concrete workflow -DONE 
2010.11.29 21:15:50.200 PST: [INFO]  Generating code for the cleanup workflow 
2010.11.29 21:15:50.323 PST: [INFO]  Generating code for the cleanup workflow -DONE 
2010.11.29 21:15:50.496 PST:    
2010.11.29 21:15:50.502 PST:   ----------------------------------------------------------------------- 
2010.11.29 21:15:50.508 PST:   File for submitting this DAG to Condor           : superdiamond-0.dag.condor.sub 
2010.11.29 21:15:50.514 PST:   Log of DAGMan debugging messages                 : superdiamond-0.dag.dagman.out 
2010.11.29 21:15:50.521 PST:   Log of Condor library output                     : superdiamond-0.dag.lib.out 
2010.11.29 21:15:50.528 PST:   Log of Condor library error messages             : superdiamond-0.dag.lib.err 
2010.11.29 21:15:50.559 PST:   Log of the life of condor_dagman itself          : superdiamond-0.dag.dagman.log 
2010.11.29 21:15:50.578 PST:    
2010.11.29 21:15:50.588 PST:   -no_submit given, not submitting DAG to Condor.  You can do this with: 
2010.11.29 21:15:50.601 PST:   "condor_submit superdiamond-0.dag.condor.sub" 
2010.11.29 21:15:50.618 PST:   ----------------------------------------------------------------------- 
2010.11.29 21:15:50.625 PST:   Submitting job(s). 
2010.11.29 21:15:50.637 PST:   Logging submit event(s). 
2010.11.29 21:15:50.642 PST:   1 job(s) submitted to cluster 1. 
2010.11.29 21:15:51.179 PST:    
2010.11.29 21:15:51.185 PST:   Your Workflow has been started and runs in base directory given below 
2010.11.29 21:15:51.191 PST:    
2010.11.29 21:15:51.197 PST:   cd /home/tutorial/pegasus-wms/dags/tutorial/pegasus/superdiamond/run0001 
2010.11.29 21:15:51.208 PST:    
2010.11.29 21:15:51.214 PST:   *** To monitor the workflow you can run *** 
2010.11.29 21:15:51.220 PST:    
2010.11.29 21:15:51.227 PST:   pegasus-status -l /home/tutorial/pegasus-wms/dags/tutorial/pegasus/superdiamond/run0001 
2010.11.29 21:15:51.234 PST:    
2010.11.29 21:15:51.240 PST:   *** To remove your workflow run *** 
2010.11.29 21:15:51.245 PST:   pegasus-remove -d 1.0 
2010.11.29 21:15:51.253 PST:   or 
2010.11.29 21:15:51.261 PST:   pegasus-remove /home/tutorial/pegasus-wms/dags/tutorial/pegasus/superdiamond/run0001 
2010.11.29 21:15:51.268 PST:    
2010.11.29 21:15:51.277 PST:   Time taken to execute is 2.745 seconds 
2010.11.29 21:15:51.277 PST: [INFO] event.pegasus.planner planner.version 3.0.0  - FINISHED 

You can track the workflow using the pegasus-status command

$ watch  pegasus-status -l /home/tutorial/pegasus-wms/dags/tutorial/pegasus/superdiamond/run0001 

After the workflow has completed you will see the black-1-f.d and black-2-f.d in the storage directory

$ ls -lh /home/tutorial/local-storage/storage/black-*


-rw-r--r-- 1 pegasus pegasus 3.6K Nov 29 21:36 /home/tutorial/local-storage/storage/black-1-f.d
-rw-r--r-- 1 pegasus pegasus 3.6K Nov 29 21:41 /home/tutorial/local-storage/storage/black-2-f.d