org.griphyn.cPlanner.transfer.refiner
Class Condor

java.lang.Object
  org.griphyn.cPlanner.transfer.AbstractRefiner
      org.griphyn.cPlanner.transfer.MultipleFTPerXFERJobRefiner
          org.griphyn.cPlanner.transfer.refiner.Condor

All Implemented Interfaces:

Refiner

public class Condor
extends MultipleFTPerXFERJobRefiner

A refiner that relies on the Condor file transfer mechanism to get the raw input data to the remote working directory. It is to be used for doing the file transfers in a condor pool, while trying to run on the local filesystem of the worker nodes.

Additionally, this will only work with local replica selector that prefers file urls from the submit host for staging.

In order to use the transfer implementation implemented by this class,

     - property pegasus.transfer.refiner must be set to
       value Condor.
     - property pegasus.selector.replica must be set to value
       Local
     - property pegasus.execute.*.filesystem.local must be set to value
       true
 

Version:

$Revision: 455 $

Author:

Karan Vahi

Field Summary

static String	DESCRIPTION A short description of the transfer refinement.
protected String	mLogMsg The string holding the logging messages
protected PoolInfoProvider	mSCHandle The handle to the Site Catalog.

Fields inherited from class org.griphyn.cPlanner.transfer.AbstractRefiner

mDAG, mLogger, mPOptions, mProps, mTPT, mTXInterImplementation, mTXStageInImplementation, mTXStageOutImplementation, mXMLStore

Fields inherited from interface org.griphyn.cPlanner.transfer.Refiner

INTER_POOL_PREFIX, REGISTER_PREFIX, STAGE_IN_PREFIX, STAGE_OUT_PREFIX

Fields inherited from interface org.griphyn.cPlanner.engine.Refiner

VERSION

Constructor Summary

Condor(ADag dag, PegasusProperties properties, PlannerOptions options)
The overloaded constructor.

Method Summary

void	addInterSiteTXNodes(SubInfo job, Collection files) Adds the inter pool transfer nodes that are required for transferring the output files of the parents to the jobs execution site.
void	addJob(SubInfo job) Add a new job to the workflow being refined.
void	addRelation(String parent, String child) Adds a new relation to the workflow being refiner.
void	addRelation(String parent, String child, String site, boolean parentNew) Adds a new relation to the workflow.
void	addStageInXFERNodes(SubInfo job, Collection files) Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.
void	addStageOutXFERNodes(SubInfo job, Collection files, ReplicaCatalogBridge rcb) Adds the stageout transfer nodes, that stage data to an output site specified by the user.
void	addStageOutXFERNodes(SubInfo job, Collection files, ReplicaCatalogBridge rcb, boolean deletedLeaf)
private TransferJob	createStageOutTransferJob(SubInfo job, Collection files, String directory, String txJobName) Constructs a condor file transfer job that handles multiple transfers.
void	done() Signals that the traversal of the workflow is done.
String	getDescription() Returns a textual description of the transfer mode.

Methods inherited from class org.griphyn.cPlanner.transfer.MultipleFTPerXFERJobRefiner

loadImplementations

Methods inherited from class org.griphyn.cPlanner.transfer.AbstractRefiner

getWorkflow, getXMLProducer, isSiteThirdParty, logConfigMessages, runTPTOnRemoteSite

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DESCRIPTION

public static final String DESCRIPTION

A short description of the transfer refinement.

See Also:

Constant Field Values

mLogMsg

protected String mLogMsg

The string holding the logging messages

mSCHandle

protected PoolInfoProvider mSCHandle

The handle to the Site Catalog. It is instantiated in this class.

Constructor Detail

Condor

public Condor(ADag dag,
              PegasusProperties properties,
              PlannerOptions options)

The overloaded constructor.

Parameters:

dag - the workflow to which transfer nodes need to be added.

properties - the PegasusProperties object containing all the properties required by Pegasus.

options - the options passed to the planner.

Method Detail

addStageInXFERNodes

public void addStageInXFERNodes(SubInfo job,
                                Collection files)

Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.

Parameters:

job - SubInfo object corresponding to the node to which the files are to be transferred to.

files - Collection of FileTransfer objects containing the information about source and destURL's.

addInterSiteTXNodes

public void addInterSiteTXNodes(SubInfo job,
                                Collection files)

Adds the inter pool transfer nodes that are required for transferring the output files of the parents to the jobs execution site. They are not supported in this case.

Parameters:

job - SubInfo object corresponding to the node to which the files are to be transferred to.

files - Collection of FileTransfer objects containing the information about source and destURL's.

addStageOutXFERNodes

public void addStageOutXFERNodes(SubInfo job,
                                 Collection files,
                                 ReplicaCatalogBridge rcb)

Adds the stageout transfer nodes, that stage data to an output site specified by the user.

Parameters:

job - SubInfo object corresponding to the node to which the files are to be transferred to.

files - Collection of FileTransfer objects containing the information about source and destURL's.

rcb - bridge to the Replica Catalog. Used for creating registration nodes in the workflow.

createStageOutTransferJob

private TransferJob createStageOutTransferJob(SubInfo job,
                                              Collection files,
                                              String directory,
                                              String txJobName)

Constructs a condor file transfer job that handles multiple transfers. The job itself is a /bin/true job that does the stageout using the transfer_input_files feature.

Parameters:

job - the SubInfo object for the job, in relation to which the transfer node is being added. Either the transfer node can be transferring this jobs input files to the execution pool, or transferring this job's output files to the output pool.

files - collection of FileTransfer objects representing the data files and staged executables to be transferred.

directory - the directory where the transfer job needs to be executed

txJobName - the name of transfer node.

Returns:

the created TransferJob.

addStageOutXFERNodes

public void addStageOutXFERNodes(SubInfo job,
                                 Collection files,
                                 ReplicaCatalogBridge rcb,
                                 boolean deletedLeaf)

Parameters:

job - SubInfo object corresponding to the node to which the files are to be transferred to.

files - Collection of FileTransfer objects containing the information about source and destURL's.

rcb - bridge to the Replica Catalog. Used for creating registration nodes in the workflow.

deletedLeaf - to specify whether the node is being added for a deleted node by the reduction engine or not. default: false

done

public void done()

Signals that the traversal of the workflow is done. This would allow the transfer mechanisms to clean up any state that they might be keeping that needs to be explicitly freed.

addJob

public void addJob(SubInfo job)

Add a new job to the workflow being refined.

Parameters:

job - the job to be added.

addRelation

public void addRelation(String parent,
                        String child)

Adds a new relation to the workflow being refiner.

Parameters:

parent - the jobname of the parent node of the edge.

child - the jobname of the child node of the edge.

addRelation

public void addRelation(String parent,
                        String child,
                        String site,
                        boolean parentNew)

Adds a new relation to the workflow. In the case when the parent is a transfer job that is added, the parentNew should be set only the first time a relation is added. For subsequent compute jobs that maybe dependant on this, it needs to be set to false.

Parameters:

parent - the jobname of the parent node of the edge.

child - the jobname of the child node of the edge.

site - the execution pool where the transfer node is to be run.

parentNew - the parent node being added, is the new transfer job and is being called for the first time.

getDescription

public String getDescription()

Returns a textual description of the transfer mode.

Returns:

a short textual description