7.10. Remote PBS Cluster using BOSCO and SSH

BOSCO enables HTCondor to submit jobs to remote PBS clusters using SSH. This section describes how to specify a site catalog entry for a site that has been configured for BOSCO job submissions.

First, the site needs to be setup for BOSCO according to the BOSCO documentation. BOSCO uses glite to submit jobs to the PBS scheduler on the remote cluster. You will also need to configure the glite installed for BOSCO on the remote system according to the documentation in the glite section in order for the mapping of Pegasus profiles to PBS job requirements to work. In particular, you will need to install the pbs_local_submit_attributes.sh and sge_local_submit_attributes.sh scripts in the correct place in the glite bin directory on the remote cluster, usually in the directory ~/bosco/glite/bin/ .

Second, to tag a site for SSH submission, the following profiles need to be specified for the site in the site catalog:

  1. pegasus profile style with value set to ssh

  2. Specify the service information as grid gateways. This should match what BOSCO provided when the cluster was set up.

An example site catalog entry for a BOSCO site looks like this:

<sitecatalog xmlns="http://pegasus.isi.edu/schema/sitecatalog"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://pegasus.isi.edu/schema/sitecatalog http://pegasus.isi.edu/schema/sc-4.0.xsd"
             version="4.0">

    <site  handle="USC_HPCC_Bosco" arch="x86_64" os="LINUX">

        <!-- Specify the service information. This should match what Bosco provided when the cluster
             was set up. -->
        <grid type="batch" contact="vahi@hpc-pegasus.usc.edu" scheduler="PBS" jobtype="compute"/>
        <grid type="batch" contact="vahi@hpc-pegasus.usc.edu" scheduler="PBS" jobtype="auxillary"/>

        <!-- Scratch directory on the cluster -->
        <directory type="shared-scratch" path="/home/rcf-40/vahi/tmp">
            <file-server operation="all" url="scp://vahi@hpc-pegasus.usc.edu/home/rcf-40/vahi/tmp"/>
        </directory>

        <!-- SSH is the style to use for Bosco SSH submits -->
        <profile namespace="pegasus" key="style">ssh</profile>

        <!--  works around bug in the HTCondor GAHP, that does not
              set the remote directory -->
        <profile namespace="pegasus" key="change.dir">true</profile>

        <!-- Job requirements should be specified using Pegasus profiles -->
        <profile namespace="pegasus" key="queue">default</profile>
        <profile namespace="pegasus" key="runtime">30</profile>

    </site>


</sitecatalog>

Note

It is recommended to have a submit node configured either as a BOSCO submit node or a vanilla HTCondor node. You cannot have HTCondor configured both as a BOSCO install and a traditional HTCondor submit node at the same time as BOSCO will override the traditional HTCondor pool in the user environment.

There is a bosco-shared-fs example in the examples directory of the distribution.

Job Requirements for the jobs can be set using the same profiles as listed here .