org.griphyn.cPlanner.transfer.implementation
Class CRFT

java.lang.Object
  org.griphyn.cPlanner.transfer.implementation.Abstract
      org.griphyn.cPlanner.transfer.implementation.AbstractMultipleFTPerXFERJob
          org.griphyn.cPlanner.transfer.implementation.CRFT

All Implemented Interfaces:

Implementation, MultipleFTPerXFERJob

public class CRFT
extends AbstractMultipleFTPerXFERJob

The implementation that creates transfer jobs referring to the c rft client executable distributed with the VDS.

The rft client is invoked on the submit host. Hence there should be an entry in the transformation catalog for logical transformation CRFT at site local. The transformation should point to the client that is distributed with RFT in GT4.

The user can tweak the options to the globus-crft client by specifying the properties in the properties files with the prefix pegasus.transfer.crft. The following table lists all the properties with their prefixes stripped off, that the user can specify in the properties files. The default value is used if the user does not specify a particular property. If a value is not specified, the particular option is not generated.

For the properties which have a default value of no default and the user not providing a value for the property, the option is not propogated further to the client underneath. In that case, it is upto the client to construct the appropriate value for that property/option.

property	default value	description
endpoint	no default (required option)	The endpoint to contact when creating a service.
concurrent	no default	The number of simultaneous transfers.
parallel	no default	The number of parallel sockets to use with each transfer.
tcp-bs	no default	specifies the size (in bytes) of the TCP buffer to be used by the underlying ftp data channels
verbose	true	to generate more verbose output, helpful for debugging.

It leads to the creation of the setup chmod jobs to the workflow, that appear as parents to compute jobs in case the transfer implementation does not preserve the X bit on the file being transferred. This is required for staging of executables as part of the workflow. The setup jobs are only added as children to the stage in jobs.

In order to use the transfer implementation implemented by this class,

        - the property pegasus.transfer.*.impl must be set to value CRFT.
 

There should be an entry in the transformation catalog with the fully qualified name as globus::crft for all the sites where workflow is run, or on the local site in case of third party transfers.

Version:

$Revision: 145 $

Author:

Karan Vahi

Field Summary

static String	CONCURRENT_KEY The key name that denotes the number of files to be transferred at any given time.
static String	CREATE_KEY The key name that denotes to create a RFT service.
static String	DERIVATION_NAME The name of the underlying derivation.
static String	DERIVATION_NAMESPACE The derivation namespace for for the transfer job.
static String	DERIVATION_VERSION The derivation version number for the transfer job.
static String	DESCRIPTION A short description of the transfer implementation.
static String	END_POINT_KEY The key name that denotes the endpoint to contact when creating a service.
private Properties	mCRFTProps The properties object holding all the RFT specific properties specified by the user in the properties file.
private String	mEndPoint The end point for the service.
static String	MONITOR_KEY The key name that denotes to monitor the request.
private static String	OPTIONS_DELIMITER The options delimiter that is prepended before all the options.
static String	PARALLEL_KEY The key name that denotes the number of parallel sockets to use for each transfer.
static String	PROPERTIES_PREFIX The prefix for all the properties this mode requires.
static String	SUBMIT_KEY The key name that denotes to start a RFT service.
static String	TCP_BUFFER_SIZE_KEY The key name that denotes the TCP buffer size in bytes.
static String	TRANSFER_FILE_KEY The key name that points to the transfer file that is containing the source and destination urls.
static String	TRANSFORMATION_NAME The name of the underlying transformation that is queried for in the Transformation Catalog.
static String	TRANSFORMATION_NAMESPACE The transformation namespace for the transfer job.
static String	TRANSFORMATION_VERSION The version number for the transfer job.
static String	VERBOSE_KEY The key name that denotes whether to do verbose or not.

Fields inherited from class org.griphyn.cPlanner.transfer.implementation.Abstract

CHANGE_XBIT_TRANSFORMATION, mDisabledChmodSites, mLocalUserProxy, mLocalUserProxyBasename, mLogger, mPOptions, mProps, mRefiner, mSCHandle, mTCHandle, NOOP_PREFIX, SET_XBIT_PREFIX, XBIT_DERIVATION_NS, XBIT_DERIVATION_VERSION, XBIT_TRANSFORMATION_NS, XBIT_TRANSFORMATION_VERSION

Fields inherited from interface org.griphyn.cPlanner.transfer.Implementation

TRANSFER_UNIVERSE, VERSION

Constructor Summary

CRFT(PegasusProperties properties, PlannerOptions options)
The overloaded constructor, that is called by the Factory to load the class.

Method Summary

boolean	addSetXBitJobs(String computeJobName, String txJobName, Collection execFiles, int transferClass) Adds the dirmanager job to the workflow, that do a chmod on the executable files that are being staged.
private String	construct(String option, String value) A helper method to generate a required argument option for the client.
TransferJob	createTransferJob(SubInfo job, Collection files, Collection execFiles, String txJobName, int jobClass) Calls out to the super class method to create the main structure of the job.
boolean	doesPreserveXBit() Returns a boolean indicating whether the transfer protocol being used by the implementation preserves the X Bit or not while staging.
protected String	generateArgumentString(TransferJob job) Constructs the arguments to the transfer executable that need to be passed to the executable referred to in this transfer mode.
protected String	getCompleteTCName() Returns the complete name for the transformation.
protected String	getDerivationName() Returns the logical name of the derivation that this implementation refers to.
protected String	getDerivationNamespace() Returns the namespace of the derivation that this implementation refers to.
protected String	getDerivationVersion() Returns the version of the derivation that this implementation refers to.
String	getDescription() Returns a textual description of the transfer implementation.
protected List	getEnvironmentVariables(String site) Returns the environment profiles that are required for the default entry to sensibly work.
TransformationCatalogEntry	getTransformationCatalogEntry(String siteHandle) Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation.
boolean	useThirdPartyTransferAlways() Return a boolean indicating whether the transfers to be done always in a third party transfer mode.
protected void	writeJumboStdIn(FileWriter writer, Collection files) Writes to a file on the submit host, that is passed to the rft-client as input.

Methods inherited from class org.griphyn.cPlanner.transfer.implementation.AbstractMultipleFTPerXFERJob

defaultTCEntry, postProcess, prepareSTDIN

Methods inherited from class org.griphyn.cPlanner.transfer.implementation.Abstract

addSetXBitJobs, addSetXBitJobs, addSetXBitJobs, applyPriority, checkAndTransferProxy, construct, createNoOPJob, createSetXBitJob, determineDisabledChmodSites, disableChmodJobCreation, getNOOPJobName, getPathToUserProxy, getPriority, getSetXBitJobName, setRefiner

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TRANSFORMATION_NAMESPACE

public static final String TRANSFORMATION_NAMESPACE

The transformation namespace for the transfer job.

See Also:

Constant Field Values

TRANSFORMATION_NAME

public static final String TRANSFORMATION_NAME

The name of the underlying transformation that is queried for in the Transformation Catalog.

See Also:

Constant Field Values

TRANSFORMATION_VERSION

public static final String TRANSFORMATION_VERSION

The version number for the transfer job.

DERIVATION_NAMESPACE

public static final String DERIVATION_NAMESPACE

The derivation namespace for for the transfer job.

See Also:

Constant Field Values

DERIVATION_NAME

public static final String DERIVATION_NAME

The name of the underlying derivation.

See Also:

Constant Field Values

DERIVATION_VERSION

public static final String DERIVATION_VERSION

The derivation version number for the transfer job.

See Also:

Constant Field Values

DESCRIPTION

public static final String DESCRIPTION

A short description of the transfer implementation.

See Also:

Constant Field Values

PROPERTIES_PREFIX

public static final String PROPERTIES_PREFIX

The prefix for all the properties this mode requires.

See Also:

Constant Field Values

END_POINT_KEY

public static final String END_POINT_KEY

The key name that denotes the endpoint to contact when creating a service.

See Also:

Constant Field Values

CREATE_KEY

public static final String CREATE_KEY

The key name that denotes to create a RFT service.

See Also:

Constant Field Values

SUBMIT_KEY

public static final String SUBMIT_KEY

The key name that denotes to start a RFT service.

See Also:

Constant Field Values

MONITOR_KEY

public static final String MONITOR_KEY

The key name that denotes to monitor the request. Makes the client block.

See Also:

Constant Field Values

TCP_BUFFER_SIZE_KEY

public static final String TCP_BUFFER_SIZE_KEY

The key name that denotes the TCP buffer size in bytes.

See Also:

Constant Field Values

VERBOSE_KEY

public static final String VERBOSE_KEY

The key name that denotes whether to do verbose or not.

See Also:

Constant Field Values

CONCURRENT_KEY

public static final String CONCURRENT_KEY

The key name that denotes the number of files to be transferred at any given time.

See Also:

Constant Field Values

PARALLEL_KEY

public static final String PARALLEL_KEY

The key name that denotes the number of parallel sockets to use for each transfer.

See Also:

Constant Field Values

TRANSFER_FILE_KEY

public static final String TRANSFER_FILE_KEY

The key name that points to the transfer file that is containing the source and destination urls.

See Also:

Constant Field Values

OPTIONS_DELIMITER

private static final String OPTIONS_DELIMITER

The options delimiter that is prepended before all the options.

See Also:

Constant Field Values

mEndPoint

private String mEndPoint

The end point for the service.

mCRFTProps

private Properties mCRFTProps

The properties object holding all the RFT specific properties specified by the user in the properties file.

Constructor Detail

CRFT

public CRFT(PegasusProperties properties,
            PlannerOptions options)

The overloaded constructor, that is called by the Factory to load the class.

Parameters:

properties - the properties object.

options - the options passed to the Planner.

Method Detail

createTransferJob

public TransferJob createTransferJob(SubInfo job,
                                     Collection files,
                                     Collection execFiles,
                                     String txJobName,
                                     int jobClass)

Calls out to the super class method to create the main structure of the job. In addition, for the CRFT adds the specific condor magic that allows for the transfer of the input file correctly to working directory.

Specified by:

createTransferJob in interface Implementation

Overrides:

createTransferJob in class AbstractMultipleFTPerXFERJob

Parameters:

job - the SubInfo object for the job, in relation to which the transfer node is being added. Either the transfer node can be transferring this jobs input files to the execution pool, or transferring this job's output files to the output pool.

files - collection of FileTransfer objects representing the data files and staged executables to be transferred.

execFiles - subset collection of the files parameter, that identifies the executable files that are being transferred.

txJobName - the name of transfer node.

jobClass - the job Class for the newly added job. Can be one of the following: stage-in stage-out inter-pool transfer

Returns:

the created TransferJob.

getEnvironmentVariables

protected List getEnvironmentVariables(String site)

Returns the environment profiles that are required for the default entry to sensibly work. Returns an empty list.

Specified by:

getEnvironmentVariables in class AbstractMultipleFTPerXFERJob

Parameters:

site - the site where the job is going to run.

Returns:

List of environment variables, else null in case where the required environment variables could not be found.

useThirdPartyTransferAlways

public boolean useThirdPartyTransferAlways()

Return a boolean indicating whether the transfers to be done always in a third party transfer mode. This always returns true, indicating transfers can only be done in a third party transfer mode. A value of false does not preclude third party transfers. They still can be done, by setting the property "pegasus.transfer.*.thirdparty.sites".

Returns:

false

See Also:

PegasusProperties.getThirdPartySites(String)

doesPreserveXBit

public boolean doesPreserveXBit()

Returns a boolean indicating whether the transfer protocol being used by the implementation preserves the X Bit or not while staging.

Returns:

boolean

addSetXBitJobs

public boolean addSetXBitJobs(String computeJobName,
                              String txJobName,
                              Collection execFiles,
                              int transferClass)

Adds the dirmanager job to the workflow, that do a chmod on the executable files that are being staged. It is empty as RFT preserves X bit permission while staging files.

Parameters:

computeJobName - the name pf the computeJob for which the files are being staged.

txJobName - the name of the transfer job that is staging the files.

execFiles - the executable files that are being staged.

transferClass - the class of transfer job

Returns:

boolean indicating whether any XBitJobs were succesfully added or not.

generateArgumentString

protected String generateArgumentString(TransferJob job)

Constructs the arguments to the transfer executable that need to be passed to the executable referred to in this transfer mode. Since the rft client is run on the submit host, the path to the input file to the rft client is given, instead of passing it through condor files. In addition , it SETS THE STDIN of the transfer job to null, as the input file is not being sent to the remote sides. There should be a generic function prepareIPFile to do this.

Specified by:

generateArgumentString in class AbstractMultipleFTPerXFERJob

Parameters:

job - the object containing the transfer node.

Returns:

the argument string

writeJumboStdIn

protected void writeJumboStdIn(FileWriter writer,
                               Collection files)
                        throws Exception

Writes to a file on the submit host, that is passed to the rft-client as input. The rft-client is always run on the submit host, and hence can access the file.

Specified by:

writeJumboStdIn in class AbstractMultipleFTPerXFERJob

Parameters:

writer - the writer to the stdin file.

files - Collection of FileTransfer objects containing the information about sourceam fin and destURL's.

Throws:

Exception

getDescription

public String getDescription()

Returns a textual description of the transfer implementation.

Returns:

a short textual description

getTransformationCatalogEntry

public TransformationCatalogEntry getTransformationCatalogEntry(String siteHandle)

Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation.

Parameters:

siteHandle - the handle of the site where the transformation is to be searched.

Returns:

the transformation catalog entry if found, else null.

getDerivationNamespace

protected String getDerivationNamespace()

Returns the namespace of the derivation that this implementation refers to.

Specified by:

getDerivationNamespace in class AbstractMultipleFTPerXFERJob

Returns:

the namespace of the derivation.

getDerivationName

protected String getDerivationName()

Returns the logical name of the derivation that this implementation refers to.

Specified by:

getDerivationName in class AbstractMultipleFTPerXFERJob

Returns:

the name of the derivation.

getDerivationVersion

protected String getDerivationVersion()

Returns the version of the derivation that this implementation refers to.

Specified by:

getDerivationVersion in class AbstractMultipleFTPerXFERJob

Returns:

the version of the derivation.

getCompleteTCName

protected String getCompleteTCName()

Returns the complete name for the transformation.

Specified by:

getCompleteTCName in class AbstractMultipleFTPerXFERJob

Returns:

the complete name.

construct

private String construct(String option,
                         String value)

A helper method to generate a required argument option for the client. It is generated only if a non null value is passed.

Parameters:

option - the long version of the option.

value - the value for the option

Returns:

the constructed string.