4.3. Resource Discovery (Site Catalog)

The Site Catalog describes the compute resources (which are often clusters) that we intend to run the workflow upon. A site is a homogeneous part of a cluster that has at least a single GRAM gatekeeper with a jobmanager-fork andjobmanager-<scheduler> interface and at least one gridftp server along with a shared file system. The GRAM gatekeeper can be either WS GRAM or Pre-WS GRAM. A site can also be a condor pool or glidein pool with a shared file system.

The Site Catalog can be described as an XML . Pegasus currently supports two schemas for the Site Catalog:

  1. XML4(Default) Corresponds to the schema described here.

  2. XML3(Deprecated) Corresponds to the schema described here

4.3.1. XML4

This is the default format for Pegasus 4.2. This format allows defining filesystem of shared as well as local type on the head node of the remote cluster as well as on the backend nodes

Figure 4.3. Schema Image of the Site Catalog XML4

Schema Image of the Site Catalog XML4

Below is an example of the XML4 site catalog

<?xml version="1.0" encoding="UTF-8"?>
<sitecatalog xmlns="http://pegasus.isi.edu/schema/sitecatalog"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://pegasus.isi.edu/schema/sitecatalog http://pegasus.isi.edu/schema/sc-4.0.xsd"
             version="4.0">

    <site  handle="local" arch="x86_64" os="LINUX">
        <directory type="shared-scratch" path="/tmp/workflows/scratch">
            <file-server operation="all" url="file:///tmp/workflows/scratch"/>
        </directory>
        <directory type="local-storage" path="/tmp/workflows/outputs">
            <file-server operation="all" url="file:///tmp/workflows/outputs"/>
        </directory>
    </site>

    <site  handle="condor_pool" arch="x86_64" os="LINUX">
        <grid type="gt5" contact="smarty.isi.edu/jobmanager-pbs" scheduler="PBS" jobtype="auxillary"/>
        <grid type="gt5" contact="smarty.isi.edu/jobmanager-pbs" scheduler="PBS" jobtype="compute"/>
        <directory type="shared-scratch" path="/lustre">
            <file-server operation="all" url="gsiftp://smarty.isi.edu/lustre"/>
        </directory>
        <replica-catalog type="LRC" url="rlsn://smarty.isi.edu"/>
    </site>

    <site  handle="staging_site" arch="x86_64" os="LINUX">
        <directory type="shared-scratch" path="/data">
            <file-server operation="put" url="scp://obelix.isi.edu/data"/>
            <file-server operation="get" url="http://obelix.isi.edu/data"/>
        </directory>
    </site>

</sitecatalog>
      

Described below are some of the entries in the site catalog.

  1. site - A site identifier.

  2. Directory - Info about filesystems Pegasus can use for storing temporary and long-term files. There are several configurations:

    • shared-scratch - This describe a scratch file systems. Pegasus will use this to store intermediate data between jobs and other temporary files.

    • local-storage - This describes the storage file systems (long term). This is the directory Pegasus will stage output files to.

    • local-scratch - This describe the scratch file systems available locally on a compute node. This parameter is not commonly used and can be left unset in most cases.

    For each of the directories, you can specify access methods. Allowed methods are put, get, and all which means both put and get. For each mehod, specify a URL including the protocol. For example, if you want share data via http using the /var/www/staging directory, you can use scp://hostname/var/www for the put element and http://hostname/staging for the get element.

  3. arch,os,osrelease,osversion, glibc - The arch/os/osrelease/osversion/glibc of the site. OSRELEASE, OSVERSION and GLIBC are optional

    ARCH can have one of the following values X86, X86_64, SPARCV7, SPARCV9, AIX, PPC.

    OS can have one of the following values LINUX,SUNOS,MACOSX. The default value for sysinfo if none specified is X86::LINUX

  4. replica-catalog - URL for a local replica catalog (LRC) to register your files in. Only used for RLS implementation of the RC. This is optional and support for RLS has been dropped in Pegasus 4.5.0 release.

  5. Profiles - One or many profiles can be attached to a site.

    One example is the environments to be set on a remote site.

To use this site catalog the follow properties need to be set:

  1. pegasus.catalog.site.file=<path to the site catalog file>

4.3.2. XML3

Warning

This format is now deprecated in favor of the XML4 format. If you are still using the File format you should convert it to XML4 format using the client pegasus-sc-converter

This is the default format for Pegasus 3.0. This format allows defining filesystem of shared as well as local type on the head node of the remote cluster as well as on the backend nodes

Figure 4.4. Schema Image of the Site Catalog XML 3

Schema Image of the Site Catalog XML 3

Below is an example of the XML3 site catalog

<sitecatalog xmlns="http://pegasus.isi.edu/schema/sitecatalog" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://pegasus.isi.edu/schema/sitecatalog 
http://pegasus.isi.edu/schema/sc-3.0.xsd" version="3.0">
  <site  handle="isi" arch="x86" os="LINUX" osrelease="" osversion="" glibc="">
      <grid  type="gt2" contact="smarty.isi.edu/jobmanager-pbs" scheduler="PBS" jobtype="auxillary"/>
      <grid  type="gt2" contact="smarty.isi.edu/jobmanager-pbs" scheduler="PBS" jobtype="compute"/>
          <head-fs>
               <scratch>
                  <shared>
                     <file-server protocol="gsiftp" url="gsiftp://skynet-data.isi.edu"
                                  mount-point="/nfs/scratch01" />
                     <internal-mount-point mount-point="/nfs/scratch01"/>
                  </shared>
               </scratch>
               <storage>
                  <shared>
                     <file-server protocol="gsiftp" url="gsiftp://skynet-data.isi.edu" 
                                  mount-point="/exports/storage01"/>
                     <internal-mount-point mount-point="/exports/storage01"/>
                  </shared>
               </storage>
          </head-fs>
      <replica-catalog  type="LRC" url="rlsn://smarty.isi.edu"/>
      <profile namespace="env" key="PEGASUS_HOME" >/nfs/vdt/pegasus</profile>
      <profile namespace="env" key="GLOBUS_LOCATION" >/vdt/globus</profile>
  </site>
</sitecatalog>

Described below are some of the entries in the site catalog.

  1. site - A site identifier.

  2. replica-catalog - URL for a local replica catalog (LRC) to register your files in. Only used for RLS implementation of the RC. This is optional and support for RLS has been dropped in Pegasus 4.5.0.

  3. File Systems - Info about filesystems mounted on the remote clusters head node or worker nodes. It has several configurations

    • head-fs/scratch - This describe the scratch file systems (temporary for execution) available on the head node

    • head-fs/storage - This describes the storage file systems (long term) available on the head node

    • worker-fs/scratch - This describe the scratch file systems (temporary for execution) available on the worker node

    • worker-fs/storage - This describes the storage file systems (long term) available on the worker node

    Each scratch and storage entry can contain two sub entries,

    • SHARED for shared file systems like NFS, LUSTRE etc.

    • LOCAL for local file systems (local to the node/machine)

    Each of the filesystems are defined by used a file-server element. Protocol defines the protocol uses to access the files, URL defines the url prefix to obtain the files from and mount-point is the mount point exposed by the file server.

    Along with this an internal-mount-point needs to defined to access the files directly from the machine without any file servers.

  4. arch,os,osrelease,osversion, glibc - The arch/os/osrelease/osversion/glibc of the site. OSRELEASE, OSVERSION and GLIBC are optional

    ARCH can have one of the following values X86, X86_64, SPARCV7, SPARCV9, AIX, PPC.

    OS can have one of the following values LINUX,SUNOS,MACOSX. The default value for sysinfo if none specified is X86::LINUX

  5. Profiles - One or many profiles can be attached to a pool.

    One example is the environments to be set on a remote pool.

To use this site catalog the follow properties need to be set:

  1. pegasus.catalog.site.file=<path to the site catalog file>

4.3.3. Site Catalog Converter pegasus-sc-converter

Pegasus 4.2 by default now parses Site Catalog format conforming to the SC schema 4.0 (XML4) available here and is explained in detail in the Catalog Properties section of Running Workflows.

Pegasus 4.2 comes with a pegasus-sc-converter that will convert users old site catalog (XML3) to the XML4 format. Sample usage is given below.

$ pegasus-sc-converter -i sample.sites.xml -I XML3 -o sample.sites.xml4 -O XML4

2010.11.22 12:55:14.169 PST:   Written out the converted file to sample.sites.xml4

To use the converted site catalog, in the properties do the following:

  1. unset pegasus.catalog.site or set pegasus.catalog.site to XML

  2. point pegasus.catalog.site.file to the converted site catalog