13.2. Profiles

13.2.1. Profile Structure Heading

All profiles are triples comprised of a namespace, a name or key, and a value. The namespace is a simple identifier. The key has only meaning within its namespace, and it's yet another identifier. There are no constraints on the contents of a value

Profiles may be represented with different syntaxes in different context. However, each syntax will describe the underlying triple.

13.2.2. Sources for Profiles

Profiles may enter the job-processing stream at various stages. Depending on the requirements and scope a profile is to apply, profiles can be associated at

  • as user property settings.

  • dax level

  • in the site catalog

  • in the transformation catalog

Unfortunately, a different syntax applies to each level and context. This section shows the different profile sources and syntaxes. However, at the foundation of each profile lies the triple of namespace, key and value.

13.2.2.1. User Profiles in Properties

Users can specify all profiles in the properties files where the property name is [namespace].key and value of the property is the value of the profile.

Namespace can be env|condor|globus|dagman|pegasus

Any profile specified as a property applies to the whole workflow i.e (all jobs in the workflow) unless overridden at the DAX level , Site Catalog , Transformation Catalog Level.

Some profiles that they can be set in the properties file are listed below

env.JAVA_HOME "/software/bin/java"

condor.periodic_release 5
condor.periodic_remove  my_own_expression
condor.stream_error true
condor.stream_output fa

globus.maxwalltime  1000
globus.maxtime      900
globus.maxcputime   10
globus.project      test_project
globus.queue        main_queue

dagman.post.arguments --test arguments
dagman.retry  4
dagman.post simple_exitcode
dagman.post.path.simple_exitcode  /bin/exitcode/exitcode.sh
dagman.post.scope all
dagman.maxpre  12
dagman.priority 13

dagman.bigjobs.maxjobs 1


pegasus.clusters.size 5

pegasus.stagein.clusters 3

13.2.2.2. Profiles in DAX

The user can associate profiles with logical transformations in DAX. Environment settings required by a job's application, or a maximum estimate on the run-time are examples for profiles at this stage.

<job id="ID000001" namespace="asdf" name="preprocess" version="1.0"
 level="3" dv-namespace="voeckler" dv-name="top" dv-version="1.0">
  <argument>-a top -T10  -i <filename file="voeckler.f.a"/>
 -o <filename file="voeckler.f.b1"/>
 <filename file="voeckler.f.b2"/></argument>
  <profile namespace="pegasus" key="walltime">2</profile>
  <profile namespace="pegasus" key="diskspace">1</profile>
  &mldr;
</job>

13.2.2.3. Profiles in Site Catalog

If it becomes necessary to limit the scope of a profile to a single site, these profiles should go into the site catalog. A profile in the site catalog applies to all jobs and all application run at the site. Commonly, site catalog profiles set environment settings like the LD_LIBRARY_PATH, or globus rsl parameters like queue and project names.

Currently, there is no tool to manipulate the site catalog, e.g. by adding profiles. Modifying the site catalog requires that you load it into your editor.

The XML version of the site catalog uses the following syntax:

<profile namespace="namespace" key="key">value</profile>
<site  handle="CCG" arch="x86_64" os="LINUX">
     <grid  type="gt5" contact="obelix.isi.edu/jobmanager-fork" scheduler="Fork" jobtype="auxillary"/>
        
     <directory type="shared-scratch" path="/shared-scratch">
            <file-server operation="all" url="gsiftp://headnode.isi.edu/shared-scratch"/>
     </directory>
     <directory type="local-storage" path="/local-storage">
            <file-server operation="all" url="gsiftp://headnode.isi.edu/local-storage"/>
     </directory> 
     <profile namespace="pegasus" key="clusters.num">1</profile>
     <profile namespace="env" key="PEGASUS_HOME">/usr</profile>        
</site>

13.2.2.4. Profiles in Transformation Catalog

Some profiles require a narrower scope than the site catalog offers. Some profiles only apply to certain applications on certain sites, or change with each application and site. Transformation-specific and CPU-specific environment variables, or job clustering profiles are good candidates. Such profiles are best specified in the transformation catalog.

Profiles associate with a physical transformation and site in the transformation catalog. The Database version of the transformation catalog also permits the convenience of connecting a transformation with a profile.

The Pegasus tc-client tool is a convenient helper to associate profiles with transformation catalog entries. As benefit, the user does not have to worry about formats of profiles in the various transformation catalog instances.

tc-client -a -P -E -p /home/shared/executables/analyze -t INSTALLED -r isi_condor -e env::GLOBUS_LOCATION=&rdquor;/home/shared/globus&rdquor;

The above example adds an environment variable GLOBUS_LOCATION to the application /home/shared/executables/analyze on site isi_condor. The transformation catalog guide has more details on the usage of the tc-client.

tr example::keg:1.0 { 

#specify profiles that apply for all the sites for the transformation 
#in each site entry the profile can be overriden 

  profile env "APP_HOME" "/tmp/myscratch"
  profile env "JAVA_HOME" "/opt/java/1.6"

  site isi {
    profile env "HELLo" "WORLD"
    profile condor "FOO" "bar"
    profile env "JAVA_HOME" "/bin/java.1.6"
    pfn "/path/to/keg"
    arch "x86"
    os "linux"
    osrelease "fc"
    osversion "4"
    type "INSTALLED"
  }

  site wind {
    profile env "CPATH" "/usr/cpath"
    profile condor "universe" "condor"
    pfn "file:///path/to/keg"
    arch "x86"
    os "linux"
    osrelease "fc"
    osversion "4"
    type "STAGEABLE"
  }
}

Most of the users prefer to edit the transformation catalog file directly in the editor.

13.2.3. Profiles Conflict Resolution

Irrespective of where the profiles are specified, eventually the profiles are associated with jobs. Multiple sources may specify the same profile for the same job. For instance, DAX may specify an environment variable X. The site catalog may also specify an environment variable X for the chosen site. The transformation catalog may specify an environment variable X for the chosen site and application. When the job is concretized, these three conflicts need to be resolved.

Pegasus defines a priority ordering of profiles. The higher priority takes precedence (overwrites) a profile of a lower priority.

  1. Transformation Catalog Profiles

  2. Site Catalog Profiles

  3. DAX Profiles

  4. Profiles in Properties

13.2.4. Details of Profile Handling

The previous sections omitted some of the finer details for the sake of clarity. To understand some of the constraints that Pegasus imposes, it is required to look at the way profiles affect jobs.

13.2.4.1. Details of env Profiles

Profiles in the env namespace are translated to a semicolon-separated list of key-value pairs. The list becomes the argument for the Condor environment command in the job's submit file.

######################################################################
# Pegasus WMS  SUBMIT FILE GENERATOR
# DAG : black-diamond, Index = 0, Count = 1
# SUBMIT FILE NAME : findrange_ID000002.sub
######################################################################
globusrsl = (jobtype=single)
environment=GLOBUS_LOCATION=/shared/globus;LD_LIBRARY_PATH=/shared/globus/lib;
executable = /shared/software/linux/pegasus/default/bin/kickstart
globusscheduler = columbus.isi.edu/jobmanager-condor
remote_initialdir = /shared/CONDOR/workdir/isi_hourglass
universe = globus
&mldr;
queue
######################################################################
# END OF SUBMIT FILE

Condor-G, in turn, will translate the environment command for any remote job into Globus RSL environment settings, and append them to any existing RSL syntax it generates. To permit proper mixing, all environment setting should solely use the env profiles, and none of the Condor nor Globus environment settings.

If kickstart starts a job, it may make use of environment variables in its executable and arguments setting.

13.2.4.2. Details of globus Profiles

Profiles in the globus Namespaces are translated into a list of paranthesis-enclosed equal-separated key-value pairs. The list becomes the value for the Condor globusrsl setting in the job's submit file:

######################################################################
# Pegasus WMS SUBMIT FILE GENERATOR
# DAG : black-diamond, Index = 0, Count = 1
# SUBMIT FILE NAME : findrange_ID000002.sub
######################################################################
globusrsl = (jobtype=single)(queue=fast)(project=nvo)
executable = /shared/software/linux/pegasus/default/bin/kickstart
globusscheduler = columbus.isi.edu/jobmanager-condor
remote_initialdir = /shared/CONDOR/workdir/isi_hourglass
universe = globus
&mldr;
queue
######################################################################
# END OF SUBMIT FILE

For this reason, Pegasus prohibits the use of the globusrsl key in the condor profile namespace.

13.2.5. The Env Profile Namespace

The env namespace allows users to specify environment variables of remote jobs. Globus transports the environment variables, and ensure that they are set before the job starts.

The key used in conjunction with an env profile denotes the name of the environment variable. The value of the profile becomes the value of the remote environment variable.

Grid jobs usually only set a minimum of environment variables by virtue of Globus. You cannot compare the environment variables visible from an interactive login with those visible to a grid job. Thus, it often becomes necessary to set environment variables like LD_LIBRARY_PATH for remote jobs.

If you use any of the Pegasus worker package tools like transfer or the rc-client, it becomes necessary to set PEGASUS_HOME and GLOBUS_LOCATION even for jobs that run locally

Table 13.1. Useful Environment Settings

Key Attributes Description

Property Key: env.PEGASUS_HOME
Profile  Key: 
PEGASUS_HOME
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

Used by auxillary jobs created by Pegasus both on remote site and local site. Should be set usually set in the Site Catalog for the sites

Property Key: env.GLOBUS_LOCATION
Profile  Key: 
GLOBUS_LOCATION
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

Used by auxillary jobs created by Pegasus both on remote site and local site. Should be set usually set in the Site Catalog for the sites

Property Key: env.LD_LIBRARY_PATH
Profile  Key: 
LD_LIBRARY_PATH
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

Point this to $GLOBUS_LOCATION/lib, except you cannot use the dollar variable. You must use the full path. Applies to both, local and remote jobs that use Globus components and should be usually set in the site catalog for the sites

Even though Condor and Globus both permit environment variable settings through their profiles, all remote environment variables must be set through the means of env profiles.

13.2.6. The Globus Profile Namespace

The globus profile namespace encapsulates Globus resource specification language (RSL) instructions. The RSL configures settings and behavior of the remote scheduling system. Some systems require queue name to schedule jobs, a project name for accounting purposes, or a run-time estimate to schedule jobs. The Globus RSL addresses all these issues.

A key in the globus namespace denotes the command name of an RSL instruction. The profile value becomes the RSL value. Even though Globus RSL is typically shown using parentheses around the instruction, the out pair of parentheses is not necessary in globus profile specifications

The table below shows some commonly used RSL instructions. For an authoritative list of all possible RSL instructions refer to the Globus RSL specification.

Table 13.2. Useful Globus RSL Instructions

Property Key Description

Property Key: globus.count
Profile  Key: 
count
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Integer

the number of times an executable is started.

Property Key: globus.jobtype
Profile  Key: 
jobtype
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

specifies how the job manager should start the remote job. While Pegasus defaults to single, use mpi when running MPI jobs.

Property Key: globus.maxcputime
Profile  Key: 
maxcputime
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Integer

the max CPU time in minutes for a single execution of a job.

Property Key: globus.maxmemory
Profile  Key: 
maxmemory
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Integer

the maximum memory in MB required for the job

Property Key: globus.maxtime
Profile  Key: 
maxtime
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Integer

the maximum time or walltime in minutes for a single execution of a job.

Property Key: globus.maxwalltime
Profile  Key: 
maxwalltime
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Integer

the maximum walltime in minutes for a single execution of a job.

Property Key: globus.minmemory
Profile  Key: 
minmemory
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Integer

the minumum amount of memory required for this job

Property Key: globus.project
Profile  Key: 
project
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

associates an account with a job at the remote end.

Property Key: globus.queue
Profile  Key: 
queue
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

the remote queue in which the job should be run. Used when remote scheduler is PBS that supports queues.

Pegasus prevents the user from specifying certain RSL instructions as globus profiles, because they are either automatically generated or can be overridden through some different means. For instance, if you need to specify remote environment settings, do not use the environment key in the globus profiles. Use one or more env profiles instead.

Table 13.3. RSL Instructions that are not permissible

Key Reason for Prohibition
arguments you specify arguments in the arguments section for a job in the DAX
directory the site catalog and properties determine which directory a job will run in.
environment use multiple env profiles instead
executable the physical executable to be used is specified in the transformation catalog and is also dependant on the gridstart module being used. If you are launching jobs via kickstart then the executable created is the path to kickstart and the application executable path appears in the arguments for kickstart
stdin you specify in the DAX for the job
stdout you specify in the DAX for the job
stderr you specify in the DAX for the job

13.2.7. The Condor Profile Namespace

The Condor submit file controls every detail how and where a job is run. The condor profiles permit to add or overwrite instructions in the Condor submit file.

The condor namespace directly sets commands in the Condor submit file for a job the profile applies to. Keys in the condor profile namespace denote the name of the Condor command. The profile value becomes the command's argument. All condor profiles are translated into key=value lines in the Condor submit file

Some of the common condor commands that a user may need to specify are listed below. For an authoritative list refer to the online condor documentation. Note: Pegasus Workflow Planner/Mapper by default specify a lot of condor commands in the submit files depending upon the job, and where it is being run.

Table 13.4. Useful Condor Commands

Property Key Description

Property Key: condor.universe
Profile  Key: 
universe
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

Pegasus defaults to either globus or scheduler universes. Set to standard for compute jobs that require standard universe. Set to vanilla to run natively in a condor pool, or to run on resources grabbed via condor glidein.

Property Key: condor.periodic_release
Profile  Key: 
periodic_release
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

is the number of times job is released back to the queue if it goes to HOLD, e.g. due to Globus errors. Pegasus defaults to 3.

Property Key: condor.periodic_remove
Profile  Key: 
periodic_remove
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

is the number of times a job is allowed to get into HOLD state before being removed from the queue. Pegasus defaults to 3.

Property Key: condor.filesystemdomain
Profile  Key: 
filesystemdomain
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

Useful for Condor glide-ins to pin a job to a remote site.

Property Key: condor.stream_error
Profile  Key: 
stream_error 
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Boolean

boolean to turn on the streaming of the stderr of the remote job back to submit host.

Property Key: condor.stream_output
Profile  Key: 
stream_output
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Boolean

boolean to turn on the streaming of the stdout of the remote job back to submit host.

Property Key: condor.priority
Profile  Key: 
priority
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

integer value to assign the priority of a job. Higher value means higher priority. The priorities are only applied for vanilla / standard/ local universe jobs. Determines the order in which a users own jobs are executed.

Property Key: condor.request_cpus
Profile  Key: 
request_cpus
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

New in Condor 7.8.0 . Number of CPU's a job requires.

Property Key: condor.request_gpus
Profile  Key: 
request_cpus
Scope       :
 TC, SC, DAX, Properties
Since       : 4.6
Type        : String

Number of GPU's a job requires.

Property Key: condor.request_memory
Profile  Key: 
request_memory
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

New in Condor 7.8.0 . Amount of memory a job requires.

Property Key: condor.request_disk
Profile  Key: 
request_disk
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

New in Condor 7.8.0 . Amount of disk a job requires.

Other useful condor keys, that advanced users may find useful and can be set by profiles are

  1. should_transfer_files

  2. transfer_output

  3. transfer_error

  4. whentotransferoutput

  5. requirements

  6. rank

Pegasus prevents the user from specifying certain Condor commands in condor profiles, because they are automatically generated or can be overridden through some different means. The table below shows prohibited Condor commands.

Table 13.5. Condor commands prohibited in condor profiles

Key Reason for Prohibition
arguments you specify arguments in the arguments section for a job in the DAX
environment use multiple env profiles instead
executable the physical executable to be used is specified in the transformation catalog and is also dependant on the gridstart module being used. If you are launching jobs via kickstart then the executable created is the path to kickstart and the application executable path appears in the arguments for kickstart

13.2.8. The Dagman Profile Namespace

DAGMan is Condor's workflow manager. While planners generate most of DAGMan's configuration, it is possible to tweak certain job-related characteristics using dagman profiles. A dagman profile can be used to specify a DAGMan pre- or post-script.

Pre- and post-scripts execute on the submit machine. Both inherit the environment settings from the submit host when pegasus-submit-dag or pegasus-run is invoked.

By default, kickstart launches all jobs except standard universe and MPI jobs. Kickstart tracks the execution of the job, and returns usage statistics for the job. A DAGMan post-script starts the Pegasus application exitcode to determine, if the job succeeded. DAGMan receives the success indication as exit status from exitcode.

If you need to run your own post-script, you have to take over the job success parsing. The planner is set up to pass the file name of the remote job's stdout, usually the output from kickstart, as sole argument to the post-script.

The table below shows the keys in the dagman profile domain that are understood by Pegasus and can be associated at a per job basis.

Table 13.6. Useful dagman Commands that can be associated at a per job basis

Property Key Description

Property Key: dagman.pre
Profile  Key: 
PRE
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

is the path to the pre-script. DAGMan executes the pre-script before it runs the job.

Property Key: dagman.pre.arguments
Profile  Key: 
PRE.ARGUMENTS
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

are command-line arguments for the pre-script, if any.

Property Key: dagman.post
Profile  Key: 
POST
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

is the postscript type/mode that a user wants to associate with a job.
  1. pegasus-exitcode - pegasus will by default associate this postscript with all jobs launched via kickstart, as long the POST.SCOPE value is not set to NONE.

  2. none -means that no postscript is generated for the jobs. This is useful for MPI jobs that are not launched via kickstart currently.

  3. any legal identifier - Any other identifier of the form ([_A-Za-z][_A-Za-z0-9]*), than one of the 2 reserved keywords above, signifies a user postscript. This allows the user to specify their own postscript for the jobs in the workflow. The path to the postscript can be specified by the dagman profile POST.PATH.[value] where [value] is this legal identifier specified. The user postscript is passed the name of the .out file of the job as the last argument on the command line.

    For e.g. if the following dagman profiles were associated with a job X

    1. POST with value user_script /bin/user_postscript

    2. POST.PATH.user_script with value /path/to/user/script

    3. POST.ARGUMENTS with value -verbose

    then the following postscript will be associated with the job X in the .dag file

    /path/to/user/script -verbose X.out where X.out contains the stdout of the job X

Property Key: dagman.post.path.[value of dagman.post]
Profile  Key: 
post.path.[value of dagman.post]
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

the path to the post script on the submit host.

Property Key: dagman.post.arguments
Profile  Key: 
POST.ARGUMENTS
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

are the command line arguments for the post script, if any.

Property Key: dagman.retry
Profile  Key: 
RETRY
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Integer
Default     : 1

is the number of times DAGMan retries the full job cycle from pre-script through post-script, if failure was detected.

Property Key: dagman.category
Profile  Key: 
CATEGORY
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

the DAGMan category the job belongs to.

Property Key: dagman.priority
Profile  Key: 
PRIORITY
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Integer

the priority to apply to a job. DAGMan uses this to select what jobs to release when MAXJOBS is enforced for the DAG.

Property Key: dagman.abort-dag-on
Profile  Key: 
ABORT-DAG-ON 
Scope       : TC, DAX,
Since       : 4.5
Type        : String

The ABORT-DAG-ON key word provides a way to abort the entire DAG if a given node returns a specific exit code (AbortExitValue). The syntax for the value of the key is AbortExitValue [RETURN DAGReturnValue] . When a DAG aborts, by default it exits with the node return value that caused the abort. This can be changed by using the optional RETURN key word along with specifying the desired DAGReturnValue


The table below shows the keys in the dagman profile domain that are understood by Pegasus and can be used to apply to the whole workflow. These are used to control DAGMan's behavior at the workflow level, and are recommended to be specified in the properties file.

Table 13.7. Useful dagman Commands that can be specified in the properties file.

Property Key Description

Property Key: dagman.maxpre
Profile  Key: 
MAXPRE
Scope       :
 Properties
Since       : 2.0
Type        : String

sets the maximum number of PRE scripts within the DAG that may be running at one time

Property Key: dagman.maxpost
Profile  Key: 
MAXPOST
Scope       :
 Properties
Since       : 2.0
Type        : String

sets the maximum number of POST scripts within the DAG that may be running at one time

Property Key: dagman.maxjobs
Profile  Key: 
MAXJOBS
Scope       :
 Properties
Since       : 2.0
Type        : String

sets the maximum number of jobs within the DAG that will be submitted to Condor at one time.

Property Key: dagman.maxidle
Profile  Key: 
MAXIDLE
Scope       :
 Properties
Since       : 2.0
Type        : String

Sets the maximum number of idle jobs allowed before HTCondor DAGMan stops submitting more jobs. Once idle jobs start to run, HTCondor DAGMan will resume submitting jobs. If the option is omitted, the number of idle jobs is unlimited.

Property Key: dagman.[CATEGORY-NAME].maxjobs
Profile  Key: 
[CATEGORY-NAME].MAXJOBS
Scope       :
 Properties
Since       : 2.0
Type        : String

is the value of maxjobs for a particular category. Users can associate different categories to the jobs at a per job basis. However, the value of a dagman knob for a category can only be specified at a per workflow basis in the properties.

Property Key: dagman.post.scope
Profile  Key: 
POST.SCOPE
Scope       :
 Properties
Since       : 2.0
Type        : String

scope for the postscripts.
  1. If set to all , means each job in the workflow will have a postscript associated with it.

  2. If set to none , means no job has postscript associated with it. None mode should be used if you are running vanilla / standard/ local universe jobs, as in those cases Condor traps the remote exitcode correctly. None scope is not recommended for grid universe jobs.

  3. If set to essential, means only essential jobs have post scripts associated with them. At present the only non essential job is the replica registration job.


13.2.9. The Pegasus Profile Namespace

The pegasus profiles allow users to configure extra options to the Pegasus Workflow Planner that can be applied selectively to a job or a group of jobs. Site selectors may use a sub-set of pegasus profiles for their decision-making.

The table below shows some of the useful configuration option Pegasus understands.

Table 13.8. Useful pegasus Profiles.

Property Key Description

Property Key: pegasus.clusters.num
Profile  Key: 
clusters.num
Scope       :
 TC, SC, DAX, Properties
Since       : 3.0
Type        : Integer

Please refer to the Pegasus Clustering Guide for detailed description. This option determines the total number of clusters per level. Jobs are evenly spread across clusters.

Property Key: pegasus.clusters.size
Profile  Key: 
clusters.size
Scope       :
 TC, SC, DAX, Properties
Since       : 3.0
Type        : Integer

Please refer to the Pegasus Clustering Guide for detailed description. This profile determines the number of jobs in each cluster. The number of clusters depends on the total number of jobs on the level.

Property Key: pegasus.job.aggregator
Profile  Key: 
job.aggregator
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Integer

Indicates the clustering executable that is used to run the clustered job on the remote site.

Property Key: pegasus.gridstart
Profile  Key: 
gridstart
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

Determines the executable for launching a job. This covers both tasks ( jobs specified by the user in the DAX) and additional jobs added by Pegasus during the planning operation. Possible values are Kickstart | NoGridStart | PegasusLite | Distribute at the moment.

Note

This profile should only be set by users if you know what you are doing. Otherwise, let Pegasus do the right thing based on your configuration.

Kickstart
By default, all jobs executed are launched using a lightweight C executable called pegasus-kickstart. This generates valuable runtime provenance information for the job as it is executed on a remote node. This information serves as the basis for the monitoring and debugging capabilities provided by Pegasus.
NoGridStart
This explicity disables the wrapping of the jobs with pegasus-kickstart. This is internally used by the planner to launch dax jobs directly. If this is set, then the information populated in the monitording database is on the basis of what is recorded in the DAGMan out file.
PegasusLite
This value is automatically associated by the Planner whenever the job runs in either nonsharedfs or condorio mode. The property pegasus.data.configuration decides whether a job is launched via PegasusLite or not. PegasusLite is a lightweight Pegasus wrapper generated for each job that allows a job to run in a nonshared file system environment and is responsible for staging in the input data and staging out the output data back to a remote staging site for the job.
Distribute

This wrapper is a HubZero specfiic wrapper that allows compute jobs that are scheduled for a local PBS cluster to be run locally on the submit host. The jobs are wrapped with a distribute wrapper that is responsible for doing the qsub and tracking of the status of the jobs in the PBS cluster.

Property Key: pegasus.gridstart.path
Profile  Key: 
gridstart.path
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : file path

Sets the path to the gridstart . This profile is best set in the Site Catalog.

Property Key: pegasus.gridstart.arguments
Profile  Key: 
gridstart.arguments
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

Sets the arguments with which GridStart is used to launch a job on the remote site.

Property Key: pegasus.stagein.clusters
Profile  Key: 
stagein.clusters
Scope       :
 TC, SC, DAX, Properties
Since       : 4.0
Type        : Integer

This key determines the maximum number of stage-in jobs that are can executed locally or remotely per compute site per workflow. This is used to configure the BalancedCluster Transfer Refiner, which is the Default Refiner used in Pegasus. This profile is best set in the Site Catalog or in the Properties file

Property Key: pegasus.stagein.local.clusters
Profile  Key: 
stagein.local.clusters
Scope       :
 TC, SC, DAX, Properties
Since       : 4.0
Type        : Integer

This key provides finer grained control in determining the number of stage-in jobs that are executed locally and are responsible for staging data to a particular remote site. This profile is best set in the Site Catalog or in the Properties file

Property Key: pegasus.stagein.remote.clusters
Profile  Key: 
stagein.remote.clusters
Scope       :
 TC, SC, DAX, Properties
Since       : 4.0
Type        : Integer

This key provides finer grained control in determining the number of stage-in jobs that are executed remotely on the remote site and are responsible for staging data to it. This profile is best set in the Site Catalog or in the Properties file

Property Key: pegasus.stageout.clusters
Profile  Key: 
stageout.clusters
Scope       :
 TC, SC, DAX, Properties
Since       : 4.0
Type        : Integer

This key determines the maximum number of stage-out jobs that are can executed locally or remotely per compute site per workflow. This is used to configure the BalancedCluster Transfer Refiner, , which is the Default Refiner used in Pegasus.

Property Key: pegasus.stageout.local.clusters
Profile  Key: 
stageout.local.clusters
Scope       :
 TC, SC, DAX, Properties
Since       : 4.0
Type        : Integer

This key provides finer grained control in determining the number of stage-out jobs that are executed locally and are responsible for staging data from a particular remote site. This profile is best set in the Site Catalog or in the Properties file

Property Key: pegasus.stageout.remote.clusters
Profile  Key: 
stageout.remote.clusters
Scope       :
 TC, SC, DAX, Properties
Since       : 4.0
Type        : Integer

This key provides finer grained control in determining the number of stage-out jobs that are executed remotely on the remote site and are responsible for staging data from it. This profile is best set in the Site Catalog or in the Properties file

Property Key: pegasus.group
Profile  Key: 
group
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

Tags a job with an arbitrary group identifier. The group site selector makes use of the tag.

Property Key: pegasus.change.dir
Profile  Key: 
change.dir
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Boolean

If true, tells kickstart to change into the remote working directory. Kickstart itself is executed in whichever directory the remote scheduling system chose for the job.

Property Key: pegasus.create.dir
Profile  Key: 
create.dir
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Boolean

If true, tells kickstart to create the the remote working directory before changing into the remote working directory. Kickstart itself is executed in whichever directory the remote scheduling system chose for the job.

Property Key: pegasus.transfer.proxy
Profile  Key: 
transfer.proxy
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Boolean

If true, tells Pegasus to explicitly transfer the proxy for transfer jobs to the remote site. This is useful, when you want to use a full proxy at the remote end, instead of the limited proxy that is transferred by CondorG.

Property Key: pegasus.style
Profile  Key: 
style
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : String

Sets the condor submit file style. If set to globus, submit file generated refers to CondorG job submissions. If set to condor, submit file generated refers to direct Condor submission to the local Condor pool. It applies for glidein, where nodes from remote grid sites are glided into the local condor pool. The default style that is applied is globus.

Property Key: pegasus.pmc_request_memory
Profile  Key: 
pmc_request_memory
Scope       :
 TC, SC, DAX, Properties
Since       : 4.2
Type        : Integer

This key is used to set the -m option for pegasus-mpi-cluster. It specifies the amount of memory in MB that a job requires. This profile is usually set in the DAX for each job.

Property Key: pegasus.pmc_request_cpus
Profile  Key: 
pmc_request_cpus
Scope       :
 TC, SC, DAX, Properties
Since       : 4.2
Type        : Integer

This key is used to set the -c option for pegasus-mpi-cluster. It specifies the number of cpu's that a job requires. This profile is usually set in the DAX for each job.

Property Key: pegasus.pmc_priority
Profile  Key: 
pmc_priority
Scope       :
 TC, SC, DAX, Properties
Since       : 4.2
Type        : Integer

This key is used to set the -p option for pegasus-mpi-cluster. It specifies the priority for a job . This profile is usually set in the DAX for each job. Negative values are allowed for priorities.

Property Key: pegasus.pmc_task_arguments
Profile  Key: 
pmc_task_arguments
Scope       :
 TC, SC, DAX, Properties
Since       : 4.2
Type        : String

The key is used to pass any extra arguments to the PMC task during the planning time. They are added to the very end of the argument string constructed for the task in the PMC file. Hence, allows for overriding of any argument constructed by the planner for any particular task in the PMC job.

Property Key: pegasus.exitcode.failuremsg
Profile  Key: 
exitcode.failuremsg
Scope       :
 TC, SC, DAX, Properties
Since       : 4.4
Type        : String

The message string that pegasus-exitcode searches for in the stdout and stderr of the job to flag failures.

Property Key: pegasus.exitcode.successmsg
Profile  Key: 
exitcode.successmsg
Scope       :
 TC, SC, DAX, Properties
Since       : 4.4
Type        : String

The message string that pegasus-exitcode searches for in the stdout and stderr of the job to determine whether a job logged it's success message or not. Note this value is used to check for whether a job failed or not i.e if this profile is specified, and pegasus-exitcode DOES NOT find the string in the job stdout or stderr, the job is flagged as failed. The complete rules for determining failure are described in the man page for pegasus-exitcode.

Property Key: pegasus.checkpoint.time
Profile  Key: 
checkpoint_time
Scope       :
 TC, SC, DAX, Properties
Since       : 4.5
Type        : Integer

the expected time in minutes for a job after which it should be sent a TERM signal to generate a job checkpoint file

Property Key: pegasus.maxwalltime
Profile  Key: 
maxwalltime
Scope       :
 TC, SC, DAX, Properties
Since       : 4.5
Type        : Integer

the maximum walltime in minutes for a single execution of a job.

Property Key: pegasus.glite.arguments
Profile  Key: 
glite.arguments
Scope       :
 TC, SC, DAX, Properties
Since       : 4.5
Type        : String

specifies the extra arguments that must appear in the local PBS generated script for a job, when running workflows on a local cluster with submissions through Glite. This is useful when you want to pass through special options to underlying LRMS such as PBS e.g. you can set value -l walltime=01:23:45 -l nodes=2 to specify your job's resource requirements.

Profile  Key: auxillary.local
Scope       :
 SC
Since       : 4.6
Type        : Boolean

indicates whether auxillary jobs associated with a compute site X, can be run on local site. This CAN ONLY be specified as a profile in the site catalog and should be set when the compute site filesystem is accessible locally on the submit host.

Property Key: pegasus.condor.arguments.quote
Profile  Key: 
condor.arguments.quote
Scope       :
 SC, Properties
Since       : 4.6
Type        : Boolean

indicates whether condor quoting rules should be applied for writing out the arguments key in the condor submit file. By default it is true unless the job is schedule to a glite style site. The value is automatically set to false for glite style sites, as condor quoting is broken in batch_gahp.

13.2.9.1. Task Resource Requirements Profiles

Startng Pegasus 4.6.0 Release, users can specify pegasus profiles to describe resources requirements for their job. The planner will automatically translate them to appropriate execution environment specific directives. For example, the profiles are automatically translated to Globus RSL keys if submitting job via CondorG to remote GRAM instances, Condor Classad keys when running in a vanilla condor pool and to appropriate shell variables for Glite that can be picked up by the local attributes.sh. The profiles are described below.

Table 13.9. Task Resource Requirement Profiles.

Property Key Description

Property Key: pegasus.runtime
Profile  Key: 
runtime
Scope       :
 TC, SC, DAX, Properties
Since       : 2.0
Type        : Long

This profile specifies the expected runtime of a job in seconds. Refer to the Pegasus Clustering Guide for description on using it for runtime clustering.

Property Key: clusters.maxruntime
Profile  Key: 
pegasus.clusters.maxruntime
Scope       :
 TC, SC, DAX, Properties
Since       : 4.0
Type        : Integer

Please refer to the Pegasus Clustering Guide for detailed description. This profile specifies the maximum runtime of a job.

Property Key: pegasus.cores
Profile  Key: 
cores
Scope       :
 TC, SC, DAX, Properties
Since       : 4.0
Type        : Integer

The total number of cores, required for a job. This is also used for accounting purposes in the database while generating statistics. It corresponds to the multiplier_factor in the job_instance table described here.

Property Key: pegasus.nodes
Profile  Key: 
nodes
Scope       :
 TC, SC, DAX, Properties
Since       : 4.6
Type        : Integer

Indicates the the number of nodes a job requires.

Property Key: pegasus.ppn
Profile  Key: 
ppn
Scope       :
 TC, SC, DAX, Properties
Since       : 4.6
Type        : Integer

Indicates the number of processors per node . This profile is best set in the Site Catalog and usually set when running workflows with MPI jobs.

Property Key: pegasus.memory
Profile  Key: 
memory
Scope       :
 TC, SC, DAX, Properties
Since       : 4.6
Type        : Long

Indicates the maximum memory a job requires in MB.

Property Key: pegasus.diskspace
Profile  Key: 
diskspace
Scope       :
 TC, SC, DAX, Properties
Since       : 4.6
Type        : Long

Indicates the maximum diskspace a job requires in MB.


The automatic translation to various execution environment specific directives is explained below. It is important, to note that execution environment specific keys take precedence over the Pegasus profile keys. For example, Globus profile key maxruntime will be preferred over Pegasus profile key runtime when running jobs via HTCondorG.

Table 13.10. Table mapping translation of Pegasus Task Requirements to corresponding execution environment keys.

Pegasus Task Resource Requirement Profile Key Corresponding Globus RSL Key Corresponding Condor Classad Key KEY in +remote_cerequirements classad for GLITE
runtime maxruntime - WALLTIME
cores count request_cpus CORES
nodes hostcount - NODES
ppn xcount - PROCS
memory maxmemory request_memory PER_PROCESS_MEMORY
diskspace - request_diskspace -

13.2.10. The Hints Profile Namespace

The hints namespace allows users to override the behavior of the Workflow Mapper during site selection. This gives you finer grained control over where a job executes and what executable it refers to. The hints namespace keys ( execution.site and pfn ) can only be specified in the DAX. It is important to note that these particular keys once specified in the DAX, cannot be overriden like other profiles.

Table 13.11. Useful Hints Profile Keys

Key Attributes Description

Property Key: N/A
Profile  Key: 
execution.site
Scope       :
 DAX
Since       : 4.5
Type        : String

the execution site where a job should be executed.

Property Key: N/A
Profile  Key: 
pfn
Scope       :
 TC, SC, DAX, Properties
Since       : 4.5
Type        : String

the physical file name to the main executable that a job refers to. Overrides any entries specified in the transformation catalog.

Property Key: hints.grid.jobtype
Profile  Key: 
grid.jobtype
Scope       :
 TC, SC, DAX, Properties
Since       : 4.5
Type        : String

applicable when submitting to remote sites via GRAM. The site catalog allows you to associate multiple jobmanagers with a GRAM site, for different type of jobs [compute, auxillary, transfer, register, cleanup ] that Pegasus generates in the executable workflow. This profile is usually used to ensure that a compute job executes on another job manager. For example, if in site catalog you have headnode.example.com/jobmanager-condor for compute jobs, and headnode.example.com/jobmanager-fork for auxillary jobs. Associating this profile and setting value to auxillary for a compute job, will cause the compute job to run on the fork jobmanager instead of the condor jobmanager.