org.griphyn.cPlanner.engine
Class ReplicaCatalogBridge

java.lang.Object
  org.griphyn.cPlanner.engine.Engine
      org.griphyn.cPlanner.engine.ReplicaCatalogBridge

public class ReplicaCatalogBridge
extends Engine

This coordinates the look up to the Replica Location Service, to determine the logical to physical mappings.

Version:

$Revision: 456 $

Author:

Karan Vahi, Gaurang Mehta

Field Summary

static String	CACHE_REPLICA_CATALOG_IMPLEMENTER The name of the Replica Catalog Implementer that serves as the source for cache files.
static String	CACHE_REPLICA_CATALOG_KEY The name of the source key for Replica Catalog Implementer that serves as cache
private ReplicaStore	mCacheStore The replica store in which we store all the results that are queried from the cache replica catalogs.
private boolean	mDefaultTCRCCreated A boolean indicating whether the attempt to create a default tc entry has happened or not.
private TransformationCatalogEntry	mDefaultTCRCEntry The default tc entry.
private ENV	mLocalEnv The namespace object holding the environment variables for local pool.
private PlannerOptions	mPOptions Contains the various options to the Planner as passed by the user at runtime.
private boolean	mRCDown A boolean variable to desingnate whether the RLI queried was down or not.
private ReplicaCatalog	mReplicaCatalog The handle to the main Replica Catalog.
private ReplicaStore	mReplicaStore The replica store in which we store all the results that are queried from the main replica catalog.
protected Set	mSearchFiles The Vector of String objects containing the logical filenames of the files whose locations are to be searched in the Replica Catalog.
private boolean	mTreatCacheAsRC A boolean indicating whether the cache file needs to be treated as a replica catalog or not.
static String	RC_DERIVATION_NAME The derivation name for the transfer jobs.
static String	RC_DERIVATION_NS The derivation namespace for the transfer jobs.
static String	RC_DERIVATION_VERSION The version number for the derivations for registration jobs.
static String	RC_TRANSFORMATION_NAME The logical name of the transformation used.
static String	RC_TRANSFORMATION_NS The transformation namespace for the regostration jobs.
static String	RC_TRANSFORMATION_VERSION The logical name of the transformation used.
static String	REPLICA_CATALOG_URL_KEY The name of the URL key for the replica catalog impelementer to be picked up.

Fields inherited from class org.griphyn.cPlanner.engine.Engine

mLogger, mLogMsg, mOutputPool, mPoolClass, mPoolFile, mPoolHandle, mProps, mRLIUrl, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE

Constructor Summary

ReplicaCatalogBridge(ADag dag, PegasusProperties properties, PlannerOptions options)
The overloaded constructor.

Method Summary

protected void	appendProperty(StringBuffer sb, String key, String value) Appends a property to the StringBuffer, in the java command line format.
void	closeConnection() To close the connection to replica services.
private TransformationCatalogEntry	defaultTCRCEntry() Returns a default TC entry to be used in case entry is not found in the transformation catalog.
void	finalize() Closes the connection to the rli.
private String	ftToRC(FileTransfer ft) Converts a FileTransfer to a RC compatible string representation.
private String	generateMappingsFile(String regJob, Collection files) Generates the registration mappings in a text file that is generated in the dax directory (the directory where all the condor submit files are generated).
private String	generateRepJobArgumentString(SiteInfo site, String regJob, Collection files) Generates the argument string to be given to the replica registration job.
private Profile	getClassPath(String home) Returns the classpath for the default rc-client entry.
protected String	getCommandLineProperties(PegasusProperties properties) Returns the properties that need to be passed to the the rc-client invocation on the command line .
ReplicaLocation	getFileLocs(String lfn) Returns all the locations as returned from the Replica Lookup Mechanism.
Set	getFilesInReplica() This returns the files for which mappings exist in the Replica Catalog.
void	initialize(ADag dag, PegasusProperties properties, PlannerOptions options) Intialises the refiner.
private void	loadCacheFiles(Set cacheFiles) Ends up loading the cache files so as to enable the lookup for the transient files created by the parent jobs.
private ENV	loadLocalEnvVariables() Reads in the environment variables into memory from the properties file and the pool catalog.
SubInfo	makeRCRegNode(String regJobName, SubInfo job, Collection files) It constructs the SubInfo object for the registration node, which registers the materialized files on the output pool in the RLS.
private ReplicaLocation	retrieveFromCache(String lfn) Retrieves a location from the cache table, that contains the contents of the cache files specified at runtime.

Methods inherited from class org.griphyn.cPlanner.engine.Engine

addVector, appendArrayList, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

RC_TRANSFORMATION_NS

public static final String RC_TRANSFORMATION_NS

The transformation namespace for the regostration jobs.

See Also:

Constant Field Values

RC_TRANSFORMATION_NAME

public static final String RC_TRANSFORMATION_NAME

The logical name of the transformation used.

See Also:

Constant Field Values

RC_TRANSFORMATION_VERSION

public static final String RC_TRANSFORMATION_VERSION

The logical name of the transformation used.

RC_DERIVATION_NS

public static final String RC_DERIVATION_NS

The derivation namespace for the transfer jobs.

See Also:

Constant Field Values

RC_DERIVATION_NAME

public static final String RC_DERIVATION_NAME

The derivation name for the transfer jobs.

See Also:

Constant Field Values

RC_DERIVATION_VERSION

public static final String RC_DERIVATION_VERSION

The version number for the derivations for registration jobs.

See Also:

Constant Field Values

CACHE_REPLICA_CATALOG_IMPLEMENTER

public static final String CACHE_REPLICA_CATALOG_IMPLEMENTER

The name of the Replica Catalog Implementer that serves as the source for cache files.

See Also:

Constant Field Values

CACHE_REPLICA_CATALOG_KEY

public static final String CACHE_REPLICA_CATALOG_KEY

The name of the source key for Replica Catalog Implementer that serves as cache

See Also:

Constant Field Values

REPLICA_CATALOG_URL_KEY

public static final String REPLICA_CATALOG_URL_KEY

The name of the URL key for the replica catalog impelementer to be picked up.

See Also:

Constant Field Values

mReplicaCatalog

private ReplicaCatalog mReplicaCatalog

The handle to the main Replica Catalog.

mPOptions

private PlannerOptions mPOptions

Contains the various options to the Planner as passed by the user at runtime.

mSearchFiles

protected Set mSearchFiles

The Vector of String objects containing the logical filenames of the files whose locations are to be searched in the Replica Catalog.

mRCDown

private boolean mRCDown

A boolean variable to desingnate whether the RLI queried was down or not. By default it is up, unless it is set to true explicitly.

mReplicaStore

private ReplicaStore mReplicaStore

The replica store in which we store all the results that are queried from the main replica catalog.

mCacheStore

private ReplicaStore mCacheStore

The replica store in which we store all the results that are queried from the cache replica catalogs.

mTreatCacheAsRC

private boolean mTreatCacheAsRC

A boolean indicating whether the cache file needs to be treated as a replica catalog or not.

mLocalEnv

private ENV mLocalEnv

The namespace object holding the environment variables for local pool.

mDefaultTCRCEntry

private TransformationCatalogEntry mDefaultTCRCEntry

The default tc entry.

mDefaultTCRCCreated

private boolean mDefaultTCRCCreated

A boolean indicating whether the attempt to create a default tc entry has happened or not.

Constructor Detail

ReplicaCatalogBridge

public ReplicaCatalogBridge(ADag dag,
                            PegasusProperties properties,
                            PlannerOptions options)

The overloaded constructor.

Parameters:

dag - the workflow that is being worked on.

properties - the properties passed to the planner.

options - the options passed to the planner at runtime.

Method Detail

initialize

public void initialize(ADag dag,
                       PegasusProperties properties,
                       PlannerOptions options)

Intialises the refiner.

Parameters:

dag - the workflow that is being worked on.

properties - the properties passed to the planner.

options - the options passed to the planner at runtime.

closeConnection

public void closeConnection()

To close the connection to replica services. This must be defined in the case where one has not done a singleton implementation. In other cases just do an empty implementation of this method.

finalize

public void finalize()

Closes the connection to the rli.

Overrides:

finalize in class Object

getFilesInReplica

public Set getFilesInReplica()

This returns the files for which mappings exist in the Replica Catalog. This should return a subset of the files which are specified in the mSearchFiles, while getting an instance to this.

Returns:

a Set of logical file names as String objects, for which logical to physical mapping exists.

See Also:

mSearchFiles

getFileLocs

public ReplicaLocation getFileLocs(String lfn)

Returns all the locations as returned from the Replica Lookup Mechanism.

Parameters:

lfn - The name of the logical file whose PFN mappings are required.

Returns:

ReplicaLocation containing all the locations for that LFN

See Also:

ReplicaLocation

makeRCRegNode

public SubInfo makeRCRegNode(String regJobName,
                             SubInfo job,
                             Collection files)

It constructs the SubInfo object for the registration node, which registers the materialized files on the output pool in the RLS. Note that the relations corresponding to this node should already have been added to the concerned DagInfo object.

Parameters:

regJobName - The name of the job which registers the files in the Replica Location Service.

job - The job whose output files are to be registered in the Replica Location Service.

files - Collection of FileTransfer objects containing the information about source and destination URLs. The destination URLs would be our PFNs.

Returns:

SubInfo corresponding to the new registration node.

defaultTCRCEntry

private TransformationCatalogEntry defaultTCRCEntry()

Returns a default TC entry to be used in case entry is not found in the transformation catalog.

Returns:

the default entry.

getClassPath

private Profile getClassPath(String home)

Returns the classpath for the default rc-client entry.

Parameters:

home - the home directory where we need to check for lib directory.

Returns:

the classpath in an environment profile.

generateRepJobArgumentString

private String generateRepJobArgumentString(SiteInfo site,
                                            String regJob,
                                            Collection files)

Generates the argument string to be given to the replica registration job. At present by default it would be picking up the file containing the mappings.

Parameters:

site - the SiteInfo object/

regJob - The name of the registration job.

files - Collection of FileTransfer objects containing the information about source and destURLs. The destination URLs would be our PFNs.

Returns:

the argument string.

getCommandLineProperties

protected String getCommandLineProperties(PegasusProperties properties)

Returns the properties that need to be passed to the the rc-client invocation on the command line . It is of the form "-Dprop1=value1 -Dprop2=value2 .."

Parameters:

properties - the properties object

Returns:

the properties list, else empty string.

appendProperty

protected void appendProperty(StringBuffer sb,
                              String key,
                              String value)

Appends a property to the StringBuffer, in the java command line format.

Parameters:

sb - the StringBuffer to append the property to.

key - the property.

value - the property value.

generateMappingsFile

private String generateMappingsFile(String regJob,
                                    Collection files)

Generates the registration mappings in a text file that is generated in the dax directory (the directory where all the condor submit files are generated). The pool value for the mapping is the output pool specified by the user when running Pegasus. The name of the file is regJob+.in

Parameters:

regJob - The name of the registration job.

files - Collection of FileTransferobjects containing the information about source and destURLs. The destination URLs would be our PFNs.

Returns:

String corresponding to the path of the the file containig the mappings in the appropriate format.

ftToRC

private String ftToRC(FileTransfer ft)

Converts a FileTransfer to a RC compatible string representation.

Parameters:

ft - the FileTransfer object

Returns:

the RC version.

retrieveFromCache

private ReplicaLocation retrieveFromCache(String lfn)

Retrieves a location from the cache table, that contains the contents of the cache files specified at runtime.

Parameters:

lfn - the logical name of the file.

Returns:

ReplicaLocation object corresponding to the entry if found, else null.

loadCacheFiles

private void loadCacheFiles(Set cacheFiles)

Ends up loading the cache files so as to enable the lookup for the transient files created by the parent jobs.

Parameters:

cacheFiles - set of paths to the cache files.

loadLocalEnvVariables

private ENV loadLocalEnvVariables()

Reads in the environment variables into memory from the properties file and the pool catalog.

Returns:

the ENV namespace object holding the environment variables.