.. _submit-directory: ======================== Submit Directory Details ======================== This chapter describes the submit directory content after Pegasus has planned a workflow. Pegasus takes in an Abstract Workflow and generates an executable workflow (DAG) in the submit directory. This document also describes the various Replica Selection Strategies in Pegasus. .. _submit-directory-layout: Layout ------ Each executable workflow is associated with a submit directory, and includes the following: #. **.dag** This is the Condor DAGMman dag file corresponding to the executable workflow generated by Pegasus. The dag file describes the edges in the DAG and information about the jobs in the DAG. Pegasus generated .dag file usually contains the following information for each job 1. The job submit file for each job in the DAG. 2. The post script that is to be invoked when a job completes. This is usually located at **$PEGASUS_HOME/bin/exitpost** and parses the kickstart record in the job's\ **.out file** and determines the exitcode. 3. JOB RETRY - the number of times the job is to be retried in case of failure. In Pegasus, the job postscript exits with a non zero exitcode if it determines a failure occurred. #. **.dag.dagman.out** When a DAG ( .dag file ) is executed by Condor DAGMan , the DAGMan writes out it's output to the **.dag.dagman.out file** . This file tells us the progress of the workflow, and can be used to determine the status of the workflow. Most of pegasus tools mine the **dagman.out** or **jobstate.log** to determine the progress of the workflows. #. **.static.bp** This file contains netlogger events that link jobs in the DAG with the jobs in the Abstract Workflow. This file is parsed by *pegasus-monitord* when a workflow starts and populates to the stampede backend. #. **.stampede.db** This is the stampede backend (a sqlite database) to which *pegasus-monitord* populates all the runtime provenance information from workflow dagman.out and job .out and .err files. #. **.replicas.db** This is the default output replica catalog (a sqlite database) to which the registration jobs populate output file locations and associated metadata. #. **.replica.store** This is a file based replica catalog, that only lists file locations are mentioned in the Abstract Workflow in the Replica Catalog Section, and is written out by the planner at mapping time. This is used to pass locations of files mentioned in the parent workflow to a sub workflow in case of hierarchical workflows. #. **.cache** This is a cache file generated by the planner that records where all the files in the workflow will be placed on the staging site, as the workflow executes. This is used in hierarchical workflows to pass locations of files in the parent workflow to when planner is invoked on the sub workflows. #. **.cache.meta** This is a file populated by *pegasus-exitcode* to record the checksum information gleamed from parsing the kickstart output present in the job.out.\* files. This is used in hierarchical workflows to pass checksum information of files in the parent workflow to when planner is invoked on the sub workflows. #. **.metadata** This is a worflow level metadata json formatted file that is written out by the planner at mapping time, and lists all metadata for workflow, jobs and files specified in the Abstract Workflow. #. **.metrics** This is a json formatted file generated by the planner, that includes some planning metrics and workflow level metrics such as different type of jobs and files. #. **.notify** This file contains all the notifications that need to be set for the workflow and the jobs in the executable workflow. The format of notify file is described :ref:`here ` . #. **.dot** Pegasus creates a dot file for the executable workflow in addition to the .dag file. This can be used to visualize the executable workflow using the dot program. #. **jobstate.log** The jobstate.log file is written out by the pegasus-monitord daemon that is launched when a workflow is submitted for execution by pegasus-run. The pegasus-monitord daemon parses the dagman.out file and writes out the jobstate.log that is easier to parse. The jobstate.log captures the various states through which a job goes during the workflow. There are other monitoring related files that are explained in the monitoring `chapter <#monitoring-files>`__. #. **braindump.yml** Contains information about pegasus version, dax file, dag file, dax label. #. **.sub** Each job in the executable workflow is associated with it's own submit file. The submit file tells Condor how to execute the job. #. **.out.00n** The stdout of the executable referred in the job submit file. In Pegasus, most jobs are launched via kickstart. Hence, this file contains the kickstart XML provenance record that captures runtime provenance on the remote node where the job was executed. n varies from 1-N where N is the JOB RETRY value in the .dag file. The exitpost executable is invoked on the .out file and it moves the .out to .out.00n so that the the job's .out files are preserved across retries. #. **.err.00n** The stderr of the executable referred in the job submit file. In case of Pegasus, mostly the jobs are launched via kickstart. Hence, this file contains stderr of kickstart. This is usually empty unless there in an error in kickstart e.g. kickstart segfaults , or kickstart location specified in the submit file is incorrect. The exitpost executable is invoked on the **.out** file and it moves the **.err to .err.00n** so that the the job's **.out** files are preserved across retries. #. **.meta** This is a file created at runtime when *pegasus-exitcode* parses the kickstart output in the job.out file. This file records metadata and checksum information for output files created by the job and recorded by *pegasus-kickstart*. .. _condor-dagman-file: HTCondor DAGMan File -------------------- The Condor DAGMan file ( .dag ) is the input to Condor DAGMan ( the workflow executor used by Pegasus ) . Pegasus generated .dag file usually contains the following information for each job: 1. The job submit file for each job in the DAG. 2. The post script that is to be invoked when a job completes. This is usually found in **$PEGASUS_HOME/bin/exitpost** and parses the kickstart record in the job's .out file and determines the exitcode. 3. JOB RETRY - the number of times the job is to be retried in case of failure. In case of Pegasus, job postscript exits with a non zero exitcode if it determines a failure occurred. 4. The pre script to be invoked before running a job. This is usually for the dax jobs in the DAX. The pre script is pegasus-plan invocation for the subdax. In the last section of the DAG file the relations between the jobs ( that identify the underlying DAG structure ) are highlighted. Sample Condor DAG File ---------------------- :: ##################################################################### # PEGASUS WMS GENERATED DAG FILE # DAG blackdiamond # Index = 0, Count = 1 ###################################################################### JOB create_dir_blackdiamond_0_isi_viz create_dir_blackdiamond_0_isi_viz.sub SCRIPT POST create_dir_blackdiamond_0_isi_viz /pegasus/bin/pegasus-exitcode \ /submit-dir/create_dir_blackdiamond_0_isi_viz.out RETRY create_dir_blackdiamond_0_isi_viz 3 JOB create_dir_blackdiamond_0_local create_dir_blackdiamond_0_local.sub SCRIPT POST create_dir_blackdiamond_0_local /pegasus/bin/pegasus-exitcode /submit-dir/create_dir_blackdiamond_0_local.out JOB pegasus_concat_blackdiamond_0 pegasus_concat_blackdiamond_0.sub JOB stage_in_local_isi_viz_0 stage_in_local_isi_viz_0.sub SCRIPT POST stage_in_local_isi_viz_0 /pegasus/bin/pegasus-exitcode \ /submit-dir/stage_in_local_isi_viz_0.out JOB chmod_preprocess_ID000001_0 chmod_preprocess_ID000001_0.sub SCRIPT POST chmod_preprocess_ID000001_0 /pegasus/bin/pegasus-exitcode \ /submit-dir/chmod_preprocess_ID000001_0.out JOB preprocess_ID000001 preprocess_ID000001.sub SCRIPT POST preprocess_ID000001 /pegasus/bin/pegasus-exitcode \ /submit-dir/preprocess_ID000001.out JOB subdax_black_ID000002 subdax_black_ID000002.sub SCRIPT PRE subdax_black_ID000002 /pegasus/bin/pegasus-plan \ -Dpegasus.user.properties=/submit-dir/./dag_1/test_ID000002/pegasus.3862379342822189446.properties\ -Dpegasus.log.*=/submit-dir/subdax_black_ID000002.pre.log \ -Dpegasus.dir.exec=app_domain/app -Dpegasus.dir.storage=duncan -Xmx1024 -Xms512\ --dir /pegasus-features/dax-3.2/dags \ --relative-dir user/pegasus/blackdiamond/run0005/user/pegasus/blackdiamond/run0005/./dag_1 \ --relative-submit-dir user/pegasus/blackdiamond/run0005/./dag_1/test_ID000002\ --basename black --sites dax_site \ --output local --force --nocleanup \ --verbose --verbose --verbose --verbose --verbose --verbose --verbose \ --verbose --monitor --deferred --group pegasus --rescue 0 \ --dax /submit-dir/./dag_1/test_ID000002/dax/blackdiamond_dax.xml JOB stage_out_local_isi_viz_0_0 stage_out_local_isi_viz_0_0.sub SCRIPT POST stage_out_local_isi_viz_0_0 /pegasus/bin/pegasus-exitcode /submit-dir/stage_out_local_isi_viz_0_0.out SUBDAG EXTERNAL subdag_black_ID000003 /Users/user/Pegasus/work/dax-3.2/black.dag DIR /duncan/test JOB clean_up_stage_out_local_isi_viz_0_0 clean_up_stage_out_local_isi_viz_0_0.sub SCRIPT POST clean_up_stage_out_local_isi_viz_0_0 /lfs1/devel/Pegasus/pegasus/bin/pegasus-exitcode \ /submit-dir/clean_up_stage_out_local_isi_viz_0_0.out JOB clean_up_preprocess_ID000001 clean_up_preprocess_ID000001.sub SCRIPT POST clean_up_preprocess_ID000001 /lfs1/devel/Pegasus/pegasus/bin/pegasus-exitcode \ /submit-dir/clean_up_preprocess_ID000001.out PARENT create_dir_blackdiamond_0_isi_viz CHILD pegasus_concat_blackdiamond_0 PARENT create_dir_blackdiamond_0_local CHILD pegasus_concat_blackdiamond_0 PARENT stage_out_local_isi_viz_0_0 CHILD clean_up_stage_out_local_isi_viz_0_0 PARENT stage_out_local_isi_viz_0_0 CHILD clean_up_preprocess_ID000001 PARENT preprocess_ID000001 CHILD subdax_black_ID000002 PARENT preprocess_ID000001 CHILD stage_out_local_isi_viz_0_0 PARENT subdax_black_ID000002 CHILD subdag_black_ID000003 PARENT stage_in_local_isi_viz_0 CHILD chmod_preprocess_ID000001_0 PARENT stage_in_local_isi_viz_0 CHILD preprocess_ID000001 PARENT chmod_preprocess_ID000001_0 CHILD preprocess_ID000001 PARENT pegasus_concat_blackdiamond_0 CHILD stage_in_local_isi_viz_0 ###################################################################### # End of DAG ###################################################################### .. _kickstart-record: Kickstart Record ---------------- Kickstart is a light weight C executable that is shipped with the pegasus worker package. All jobs are launced via Kickstart on the remote end, unless explicitly disabled at the time of running pegasus-plan. Kickstart does not work with: 1. Condor Standard Universe Jobs 2. MPI Jobs Pegasus automatically disables kickstart for the above jobs. Kickstart captures useful runtime provenance information about the job launched by it on the remote note, and puts in an XML record that it writes to its own stdout. The stdout appears in the workflow submit directory as .out.00n . The following information is captured by kickstart and logged: 1. The exitcode with which the job it launched exited. 2. The duration of the job 3. The start time for the job 4. The node on which the job ran 5. The stdout and stderr of the job 6. The arguments with which it launched the job 7. The environment that was set for the job before it was launched. 8. The machine information about the node that the job ran on Amongst the above information, the dagman.out file gives a coarser grained estimate of the job duration and start time. Reading a Kickstart Output File ------------------------------- Starting with Pegasus 5.0 *pegasus-kickstart* now writes out the runtime provenance as a YAML document instead of the earlier XML formatted document. The kickstart file below has the following fields highlighted: 1. The host on which the job executed and the ipaddress of that host 2. The duration and start time of the job. The time here is in reference to the clock on the remote node where the job is executed. 3. The exitcode with which the job executed 4. The arguments with which the job was launched. 5. The directory in which the job executed on the remote site 6. The stdout of the job 7. The stderr of the job 8. The environment of the job .. tabs:: .. tab:: YAML .. code-block:: yaml :emphasize-lines: 3,4,11-12,55-64,167-201 - invocation: True version: 3.0 start: 2020-06-12T22:25:51.876-07:00 duration: 60.039 transformation: "diamond::preprocess:4.0" derivation: "ID0000001" resource: "CCG" wf-label: "blackdiamond" wf-stamp: "2020-06-12T22:24:09-07:00" interface: eth0 hostaddr: 128.9.36.72 hostname: compute-2.isi.edu pid: 10187 uid: 579 user: ptesting gid: 100 group: users umask: 0o0022 mainjob: start: 2020-06-12T22:25:51.913-07:00 duration: 60.002 pid: 10188 usage: utime: 59.993 stime: 0.002 maxrss: 1312 minflt: 394 majflt: 0 nswap: 0 inblock: 0 outblock: 16 msgsnd: 0 msgrcv: 0 nsignals: 0 nvcsw: 2 nivcsw: 326 status: raw: 0 regular_exitcode: 0 executable: file_name: /var/lib/condor/execute/dir_9997/pegasus.nInvqOjMu/diamond-preprocess-4_0 mode: 0o100755 size: 82976 inode: 369207696 nlink: 1 blksize: 4096 blocks: 168 mtime: 2020-06-12T22:25:51-07:00 atime: 2020-06-12T22:25:51-07:00 ctime: 2020-06-12T22:25:51-07:00 uid: 579 user: ptesting gid: 100 group: users argument_vector: - -a - preprocess - -T - 60 - -i - f.a - -o - f.b1 - f.b2 procs: jobids: condor: 9774913.0 gram: https://obelix.isi.edu:49384/16866322196481424206/5750061617434002842/ cwd: /var/lib/condor/execute/dir_9997/pegasus.nInvqOjMu usage: utime: 0.004 stime: 0.034 maxrss: 816 minflt: 1358 majflt: 1 nswap: 0 inblock: 544 outblock: 0 msgsnd: 0 msgrcv: 0 nsignals: 0 nvcsw: 4 nivcsw: 3 machine: page-size: 4096 uname_system: linux uname_nodename: compute-2.isi.edu uname_release: 3.10.0-1062.4.1.el7.x86_64 uname_machine: x86_64 ram_total: 7990140 ram_free: 3355064 ram_shared: 0 ram_buffer: 0 swap_total: 0 swap_free: 0 cpu_count: 4 cpu_speed: 2600 cpu_vendor: GenuineIntel cpu_model: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz load_min1: 0.02 load_min5: 0.06 load_min15: 0.06 procs_total: 215 procs_running: 1 procs_sleeping: 214 procs_vmsize: 42446148 procs_rss: 1722380 task_total: 817 task_running: 1 task_sleeping: 816 files: f.b2: lfn: "f.b2" file_name: /var/lib/condor/execute/dir_9997/pegasus.nInvqOjMu/f.b2 mode: 0o100644 size: 114 inode: 369207699 nlink: 1 blksize: 4096 blocks: 8 mtime: 2020-06-12T22:25:51-07:00 atime: 2020-06-12T22:25:51-07:00 ctime: 2020-06-12T22:25:51-07:00 uid: 579 user: ptesting gid: 100 group: users output: True sha256: deac67f380112ecfa4b65879846a5f27abd64c125c25f8958cb1be44decf567f checksum_timing: 0.019 f.b1: lfn: "f.b1" file_name: /var/lib/condor/execute/dir_9997/pegasus.nInvqOjMu/f.b1 mode: 0o100644 size: 114 inode: 369207698 nlink: 1 blksize: 4096 blocks: 8 mtime: 2020-06-12T22:25:51-07:00 atime: 2020-06-12T22:25:51-07:00 ctime: 2020-06-12T22:25:51-07:00 uid: 579 user: ptesting gid: 100 group: users output: True sha256: deac67f380112ecfa4b65879846a5f27abd64c125c25f8958cb1be44decf567f checksum_timing: 0.018 stdin: file_name: /dev/null mode: 0o20666 size: 0 inode: 1034 nlink: 1 blksize: 4096 blocks: 0 mtime: 2019-10-29T08:35:24-07:00 atime: 2019-10-29T08:35:24-07:00 ctime: 2019-10-29T08:35:24-07:00 uid: 0 user: root gid: 0 group: root stdout: temporary_name: /var/lib/condor/execute/dir_9997/ks.out.1uMt3U descriptor: 3 mode: 0o100600 size: 0 inode: 302035961 nlink: 1 blksize: 4096 blocks: 0 mtime: 2020-06-12T22:25:51-07:00 atime: 2020-06-12T22:25:51-07:00 ctime: 2020-06-12T22:25:51-07:00 uid: 579 user: ptesting gid: 100 group: users data_truncated: false data: | Tue Oct 6 15:25:25 PDT 2020 stderr: temporary_name: /var/lib/condor/execute/dir_9997/ks.err.ict5LD descriptor: 4 mode: 0o100600 size: 0 inode: 302035962 nlink: 1 blksize: 4096 blocks: 0 mtime: 2020-06-12T22:25:51-07:00 atime: 2020-06-12T22:25:51-07:00 ctime: 2020-06-12T22:25:51-07:00 uid: 579 user: ptesting gid: 100 group: users metadata: temporary_name: /var/lib/condor/execute/dir_9997/ks.meta.TplHum descriptor: 5 mode: 0o100600 size: 0 inode: 302035963 nlink: 1 blksize: 4096 blocks: 0 mtime: 2020-06-12T22:25:51-07:00 atime: 2020-06-12T22:25:51-07:00 ctime: 2020-06-12T22:25:51-07:00 uid: 579 user: ptesting gid: 100 group: users .. tab:: XML .. code-block:: shell 23212F7573722F62696E2F656E762070 --create --dir /u/ac/vahi/globus-test/EXEC/vahi/pegasus/scb/run0001 /u/ac/vahi/globus-test/EXEC 2009-01-30T19:17:41.157-06:00 #1 SMP Mon Jan 21\ 13:29:51 UTC 2008 2009-01-15T10:19:50.283-06:00 mkdir finished successfully. 7F454C46020101000000000000000000 https://co-login.ncsa.uiuc.edu:50001/27456/1233364659/ URLx-nexus://co-login.ncsa.uiuc.edu:50002/ /usr/local/prews-gram-4.0.7-r1/ .... unlimited unlimited unlimited .... .. note:: *pegasus-kickstart* writes out the job environment in case job exits with failure (non zero exitcode). To see job environment for a successful job, pass -f option to *pegasus-kickstart*. .. _jobstate-log-file: Jobstate.Log File ----------------- The jobstate.log file logs the various states that a job goes through during workflow execution. It is created by the **pegasus-monitord** daemon that is launched when a workflow is submitted to Condor DAGMan by pegasus-run. **pegasus-monitord** parses the dagman.out file and writes out the jobstate.log file, the format of which is more amenable to parsing. .. note:: The jobstate.log file is not created if a user uses condor_submit_dag to submit a workflow to Condor DAGMan. The jobstate.log file can be created after a workflow has finished executing by running **pegasus-monitord** on the .dagman.out file in the workflow submit directory. Below is a snippet from the jobstate.log for a single job executed via condorg: :: 1239666049 create_dir_blackdiamond_0_isi_viz SUBMIT 3758.0 isi_viz - 1 1239666059 create_dir_blackdiamond_0_isi_viz EXECUTE 3758.0 isi_viz - 1 1239666059 create_dir_blackdiamond_0_isi_viz GLOBUS_SUBMIT 3758.0 isi_viz - 1 1239666059 create_dir_blackdiamond_0_isi_viz GRID_SUBMIT 3758.0 isi_viz - 1 1239666064 create_dir_blackdiamond_0_isi_viz JOB_TERMINATED 3758.0 isi_viz - 1 1239666064 create_dir_blackdiamond_0_isi_viz JOB_SUCCESS 0 isi_viz - 1 1239666064 create_dir_blackdiamond_0_isi_viz POST_SCRIPT_STARTED - isi_viz - 1 1239666069 create_dir_blackdiamond_0_isi_viz POST_SCRIPT_TERMINATED 3758.0 isi_viz - 1 1239666069 create_dir_blackdiamond_0_isi_viz POST_SCRIPT_SUCCESS - isi_viz - 1 Each entry in jobstate.log has the following: 1. The ISO timestamp for the time at which the particular event happened. 2. The name of the job. 3. The event recorded by DAGMan for the job. 4. The condor id of the job in the queue on the submit node. 5. The pegasus site to which the job is mapped. 6. The job time requirements from the submit file. 7. The job submit sequence for this workflow. .. table:: The job lifecycle when executed as part of the workflow ========================================== ====================================================================================================== **STATE/EVENT** **DESCRIPTION** SUBMIT job is submitted by condor schedd for execution. EXECUTE condor schedd detects that a job has started execution. GLOBUS_SUBMIT the job has been submitted to the remote resource. It's only written for GRAM jobs (i.e. gt2 and gt4). GRID_SUBMIT same as GLOBUS_SUBMIT event. The ULOG_GRID_SUBMIT event is written for all grid universe jobs./ JOB_TERMINATED job terminated on the remote node. JOB_SUCCESS job succeeded on the remote host, condor id will be zero (successful exit code). JOB_FAILURE job failed on the remote host, condor id will be the job's exit code. POST_SCRIPT_STARTED post script started by DAGMan on the submit host, usually to parse the kickstart output POST_SCRIPT_TERMINATED post script finished on the submit node. POST_SCRIPT_SUCCESS \| POST_SCRIPT_FAILURE post script succeeded or failed. ========================================== ====================================================================================================== There are other monitoring related files that are explained in the monitoring `chapter <#monitoring-files>`__. .. _submit-directory-delays: Pegasus Workflow Job States and Delays -------------------------------------- The various job states that a job goes through ( as caputured in the dagman.out and jobstate.log file) during it's lifecycle are illustrated below. The figure below highlights the various local and remote delays during job lifecycle. |image0| .. _braindump-file: Braindump File -------------- The braindump file is created per workflow in the submit file and contains metadata about the workflow. .. table:: Information Captured in Braindump File ================= =================================================================================================== **KEY** **DESCRIPTION** user the username of the user that ran pegasus-plan grid_dn the Distinguished Name in the proxy submit_hostname the hostname of the submit host root_wf_uuid the workflow uuid of the root workflow wf_uuid the workflow uuid of the current workflow i.e the one whose submit directory the braindump file is. dax the path to the dax file dax_label the label attribute in the adag element of the dax dax_index the index in the dax. dax_version the version of the DAX schema that DAX referred to. pegasus_wf_name the workflow name constructed by pegasus when planning timestamp the timestamp when planning occured basedir the base submit directory submit_dir the full path for the submit directory properties the full path to the properties file in the submit directory planner the planner used to construct the executable workflow. always pegasus planner_version the versions of the planner pegasus_build the build timestamp planner_arguments the arguments with which the planner is invoked. jsd the path to the jobstate file rundir the rundir in the numbering scheme for the submit directories pegasushome the root directory of the pegasus installation vogroup the vo group to which the user belongs to. Defaults to pegasus condor_log the full path to condor common log in the submit directory notify the notify file that contains any notifications that need to be sent for the workflow. dag the basename of the dag file created type the type of executable workflow. Can be dag \| shell\| pmc ================= =================================================================================================== A Sample Braindump File is displayed below: :: user vahi grid_dn null submit_hostname obelix root_wf_uuid a4045eb6-317a-4710-9a73-96a745cb1fe8 wf_uuid a4045eb6-317a-4710-9a73-96a745cb1fe8 dax /data/scratch/vahi/examples/synthetic-scec/Test.dax dax_label Stampede-Test dax_index 0 dax_version 3.3 pegasus_wf_name Stampede-Test-0 timestamp 20110726T153746-0700 basedir /data/scratch/vahi/examples/synthetic-scec/dags submit_dir /data/scratch/vahi/examples/synthetic-scec/dags/vahi/pegasus/Stampede-Test/run0005 properties pegasus.6923599674234553065.properties planner /data/scratch/vahi/software/install/pegasus/default/bin/pegasus-plan planner_version 3.1.0cvs pegasus_build 20110726221240Z planner_arguments "--conf ./conf/properties --dax Test.dax --sites local --output local --dir dags --force --submit " jsd jobstate.log rundir run0005 pegasushome /data/scratch/vahi/software/install/pegasus/default vogroup pegasus condor_log Stampede-Test-0.log notify Stampede-Test-0.notify dag Stampede-Test-0.dag type dag .. _static-bp-file: Pegasus static.bp File ---------------------- Pegasus creates a workflow.static.bp file that links jobs in the DAG with the jobs in the DAX. The contents of the file are in netlogger format. The purpose of this file is to be able to link an invocation record of a task to the corresponding job in the DAX The workflow is replaced by the name of the workflow i.e. same prefix as the .dag file In the file there are five types of events: - task.info This event is used to capture information about all the tasks in the DAX( abstract workflow) - task.edge This event is used to capture information about the edges between the tasks in the DAX ( abstract workflow ) - job.info This event is used to capture information about the jobs in the DAG ( executable workflow generated by Pegasus ) - job.edge This event is used to capture information about edges between the jobs in the DAG ( executable workflow ). - wf.map.task_job This event is used to associate the tasks in the DAX with the corresponding jobs in the DAG. .. |image0| image:: ../images/Pegasus_Job_State_Delay.jpg :width: 100.0%