11.5. Optimizing Data Transfers

Issue: When it comes to data transfers, Pegasus ships with a default configuration which is trying to strike a balance between performance and aggressiveness. We obviously want data transfers to be as quick as possibly, but we also do not want our transfers to overwhelm data services and systems.

Solution: Starting 4.8.0 release, the default configuration of Pegasus now adds transfer jobs and cleanup jobs based on the number of jobs at a particular level of the workflow. For example, for every 10 compute jobs on a level of a workflow, one data transfer job( stage-in and stage-out) is created. The default configuration also sets how many threads such a pegasus-transfer job can spawn. Cleanup jobs are similarly constructed with an internal ratio of 5.

Additionally, Pegasus makes use of DAGMan categories and associates the following default values with the transfer and cleanup jobs.

Table 11.3. Default Category names associated by Pegasus

DAGMan Category Name Auxillary Job applied to. Default Value Assigned in generated DAG file

stage-in 

data stage-in jobs 10

stage-out

data stage-out jobs 10

stage-inter

inter site data transfer jobs -

cleanup

data cleanup jobs 4

registration 

registration jobs 1 (for file based RC)


See Job Throttling for details on how to set these values.

Information on how to control manully the maxinum number of stagein and stageout jobs can be found in the Data Movement Nodes section.

How to control the number of threads pegasus-transfer can use depends on if you want to control standard transfer jobs, or PegasusLite. For the former, see the pegasus.transfer.threads property, and for the latter the pegasus.transfer.lite.threads property.