9. CWL Support

The Common Workflow Language (CWL) is a standardized way of describing workflow pipelines and tools. Users with existing CWL workflows can use the pegasus-cwl-converter to convert their workflows from CWL into Pegasus’s native format. Once converted, pegasus-plan can be used to plan, and submit the workflow for execution.

A Simple Example

../_images/examples-diamond-clear.png

Consider the workflow depicted above. The CWL tab below illustrates what this workflow might look like when represented using CWL. The Pegasus tab illustrates the resulting translation from CWL to Pegasus’s native format.

While there are inherent similarities between the two formats, this is not a one to one mapping. As such, some information needs to be provided to the pegasus-cwl-converter in order for the translation to be done correctly (see highlighted lines in the Pegasus tab).

This additional information is as follows:

  • For each transformation:

    • What site does it reside on? (e.g. local, condorpool, etc)

    • Can this transformation be staged to another site?

  1cwlVersion: v1.1
  2class: Workflow
  3inputs:
  4    f.a: File
  5outputs:
  6    final_output:
  7        type: File
  8        outputSource: analyze/of
  9steps:
 10    preprocess:
 11        run:
 12            cwlVersion: v1.1
 13            class: CommandLineTool
 14            baseCommand: $PEGASUS_LOCAL_BIN_DIR/pegasus-keg
 15            arguments: ["-a", "preprocess", "-T60", "-o", "f.b1", "-o", "f.b2"]
 16            requirements:
 17                DockerRequirement:
 18                    dockerPull: opensciencegrid/osgvo-el7
 19            inputs:
 20                if:
 21                    type: File
 22                    inputBinding:
 23                        prefix: -i
 24                        separate: true
 25                        position: 0
 26            outputs:
 27                of1:
 28                    type: File
 29                    outputBinding:
 30                        glob: f.b1
 31
 32                of2:
 33                    type: File
 34                    outputBinding:
 35                        glob: f.b2
 36
 37        in:
 38            if: f.a
 39        out: [of1, of2]
 40
 41    findrange1:
 42        run:
 43            cwlVersion: v1.1
 44            class: CommandLineTool
 45            baseCommand: $PEGASUS_LOCAL_BIN_DIR/pegasus-keg
 46            arguments: ["-a", "findrange", "-T60", "-o", "f.c1"]
 47            requirements:
 48                DockerRequirement:
 49                    dockerPull: opensciencegrid/osgvo-el7
 50            inputs:
 51                if:
 52                    type: File
 53                    inputBinding:
 54                        prefix: -i
 55                        separate: true
 56                        position: 0
 57
 58            outputs:
 59                of:
 60                    type: File
 61                    outputBinding:
 62                        glob: f.c1
 63        in:
 64            if: preprocess/of1
 65        out: [of]
 66
 67    findrange2:
 68        run:
 69            cwlVersion: v1.1
 70            class: CommandLineTool
 71            baseCommand: $PEGASUS_LOCAL_BIN_DIR/pegasus-keg
 72            arguments: ["-a", "findrange", "-T60", "-o", "f.c2"]
 73            requirements:
 74                DockerRequirement:
 75                    dockerPull: opensciencegrid/osgvo-el7
 76            inputs:
 77                if:
 78                    type: File
 79                    inputBinding:
 80                        prefix: -i
 81                        separate: true
 82                        position: 0
 83
 84            outputs:
 85                of:
 86                    type: File
 87                    outputBinding:
 88                        glob: f.c2
 89        in:
 90            if: preprocess/of2
 91        out: [of]
 92
 93    analyze:
 94        run:
 95            cwlVersion: v1.1
 96            class: CommandLineTool
 97            baseCommand: $PEGASUS_LOCAL_BIN_DIR/pegasus-keg
 98            arguments: ["-a", "analyze", "-T60", "-o", "f.d"]
 99            requirements:
100                DockerRequirement:
101                    dockerPull: opensciencegrid/osgvo-el7
102            inputs:
103                if1:
104                    type: File
105                    inputBinding:
106                        prefix: -i
107                        separate: true
108                        position: 0
109
110                if2:
111                    type: File
112                    inputBinding:
113                        position: 1
114
115            outputs:
116                of:
117                    type: File
118                    outputBinding:
119                        glob: f.d
120        in:
121            if1: findrange1/of
122            if2: findrange2/of
123        out: [of]

Using pegasus-cwl-converter

Using the pegasus-cwl-converter, how can we convert the above CWL workflow into Pegasus’s native format?

First, we need to create a file tr_spec.yml, which contains Pegasus specific information about the transformations (executables) used in the workflow.

pegasus-keg:
    site: local
    is_stageable: True

Next, we need a file that specifies where the initial input files to the workflow are physically located. For CWL workflows, input file specifications are typically specified in a separate YAML file. For the CWL workflow above, lets say that this file is called input_file_specs.yml and contains the following contents:

f.a:
    class: File
    path: /data/f.a

Using the following three files:

  1. CWL workflow file (wf.cwl)

  2. Workflow inputs file (input_file_specs.yml)

  3. Pegasus specific information about the executables (tr_specs.yml)

The converter can be invoked as:

pegasus-cwl-converter workflow.cwl input.yml tr_specs.yml  pegasus_workflow.yml

The resulting Pegasus workflow file pegasus_workflow.yml is depicted above in the Pegasus tab.

Note

pegasus-cwl-converter works on a subset of the CWL specification. If the conversion does not work for your workflow, reach out to us and we can assist you in getting your workflow up and running.