7.13. Titan Using Glite

Titan is part of Oak Ridge Leadership Computing Facilities (OLCF) and offers hybrid computing resources (CPUs and GPUs) to scientists since 2012.

In order to submit to Titan, a Titan login node or a system that has access to the Lustre filesystem and the batch scheduler (eg. OLCF's Kubernetes Deployment), must be used as the submit node. Submission style must be Pegasus Glite and an example site calatog entry looks like this:

<sitecatalog xmlns="http://pegasus.isi.edu/schema/sitecatalog"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://pegasus.isi.edu/schema/sitecatalog http://pegasus.isi.edu/schema/sc-4.1.xsd"
            version="4.1">

    <site handle="local" arch="x86_64" os="LINUX">
        <directory type="shared-scratch" path="/lustre/atlas/scratch/user/workflow-dir/scratch"/>
            <file-server operation="all" url="file:///lustre/atlas/scratch/user/workflow-dir/scratch"/>
        </directory>
        <directory type="shared-storage" path="/lustre/atlas/scratch/user/workflow-dir/output/">
            <file-server operation="all" url="file:///lustre/atlas/scratch/user/workflow-dir/output"/>
        </directory>
    </site>

    <site handle="titan" arch="x86_64" os="LINUX">
        <directory type="shared-scratch" path="/lustre/atlas/scratch/user/titan/scratch">
            <file-server operation="all" url="file:///lustre/atlas/scratch/user/titan/scratch"/>
        </directory>

        <profile namespace="pegasus" key="style">glite</profile>
        <profile namespace="condor" key="grid_resource">batch pbs</profile>

        <profile namespace="pegasus" key="queue">titan</profile>
        <profile namespace="pegasus" key="auxillary.local">true</profile>

        <profile namespace="env" key="PEGASUS_HOME">/lustre/atlas/world-shared/csc320/SOFTWARE/install/pegasus/default</profile>
        <profile namespace="pegasus" key="runtime">1800</profile>
        <profile namespace="pegasus" key="nodes">1</profile>
        <profile namespace="pegasus" key="project">CSC320</profile>
    </site>
</sitecatalog>
    

  1. pegasus profile style with value set to glite

  2. condor profile grid_resource with value set to batch pbs

  3. pegasus profile queue is mandatory and should be set to titan

  4. pegasus profile runtime is mandatory and should be set in sites or transformation catalog

  5. pegasus profile nodes is mandatory and should be set in sites or transformation catalog

  6. pegasus profile project must be set to the project name your jobs run under

Note

pegasus profile cores is incompatible with Titan's PBS submissions.