org.griphyn.cPlanner.code.generator.condor
Class CondorGenerator

java.lang.Object
  org.griphyn.cPlanner.code.generator.Abstract
      org.griphyn.cPlanner.code.generator.condor.CondorGenerator

All Implemented Interfaces:

CodeGenerator

Direct Known Subclasses:

HashedFile

public class CondorGenerator
extends Abstract

This class generates the condor submit files for the DAG which has to be submitted to the Condor DagMan.

Version:

$Revision: 461 $

Author:

Gaurang Mehta, Karan Vahi

Field Summary

protected ADag	mConcreteWorkflow The workflow for which the code has to be generated.
protected PrintWriter	mDagWriter The file handle to the .dag file.
protected boolean	mDone A boolean indicating whether the files have been generated or not.
protected GridStartFactory	mGridStartFactory The handle to the GridStart Factory.
protected boolean	mInitializeGridStart A boolean indicating whether grid start has been initialized or not.
protected LogManager	mLogger The LogManager object which is used to log all the messages.
protected String	mPoolClass Specifies the implementing class for the pool interface.
private PoolInfoProvider	mPoolHandle Handle to the pool provider.
private Map	mProjectMap Maps pool handles to project names.
private Map	mQueueMap Maps pool handles to queue names.
static String	mSeparator The nice separator, define once, use often.
protected CondorStyleFactory	mStyleFactory Handle to the Style factory, that is used for this workflow.
protected TransformationCatalog	mTCHandle Handle to the Transformation Catalog.
protected String	mTempLogFile The name of the log file in the /tmp directory
private Map	mWalltimeMap Maps pool handles to walltimes for that pool.

Fields inherited from class org.griphyn.cPlanner.code.generator.Abstract

mBag, mPOptions, mProps, mSubmitFileDir

Fields inherited from interface org.griphyn.cPlanner.code.CodeGenerator

VERSION

Constructor Summary

CondorGenerator()
The default constructor.

Method Summary

protected void	applyStyle(SubInfo job, PrintWriter writer) Applies a submit file style to the job, according to the fact whether the job has to be submitted directly to condor or to a remote jobmanager via CondorG and GRAM.
protected void	concreteDagEmpty(String filename, ADag dag) A callout method that dictates what needs to be done in case the concrete plan that is generated is empty.
private Map	constructMap(String propValue) Constructs a map with the numbers/values to be passed in the RSL handle for certain pools.
void	generateCode(ADag dag) Generates the code for the concrete workflow in Condor DAGMAN and CondorG input format.
void	generateCode(ADag dag, SubInfo job) Generates the code (condor submit file) for a single job.
protected boolean	generateLogFileSymlink(String logFile, String symlink) This method generates a symlink to the actual log file written in the local temp directory.
protected String	getDAGFilename(ADag dag, String suffix) Returns the name of the file on the basis of the metadata associated with the DAG.
protected String	getDAGMANOutFilename(ADag dag) Returns the basename of the file, that contains the output of the dagman while running the dag generated for the workflow.
protected String	getLogFileName() The name of the condor log file that is used for the dag that is being written into the condor formal.
protected String	getLogFileSymlinkName(ADag dag) Returns the path to the symlink that is created to the original log file in the temporary directory.
private String	gridstart(PrintWriter writer, SubInfo job, boolean isGlobusJob) This function creates the stdio handling with and without gridstart.
protected void	handleCondorVarForJob(SubInfo sinfo) It updates/adds the condor variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally.
protected void	handleEnvVarForJob(SubInfo sinfo) It updates/adds the environment variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally.
protected void	handleGlobusRSLForJob(SubInfo sinfo) It updates/adds the the Globus RSL parameters got through the dax that are in SubInfo object.
protected void	handleTransferOfExecutable(SubInfo sinfo) It changes the paths to the executable depending on whether we want to transfer the executable or not.
void	initialize(PegasusBag bag) Initializes the Code Generator implementation.
protected void	initializeDagFileWriter(String filename, DagInfo dinfo) Initializes the file handler to the dag file and writes the header to it.
protected void	printDagString(String str) Writes a string to the dag file.
void	reset() Resets the Code Generator implementation.
boolean	startMonitoring() Starts monitoring of the workflow by invoking a workflow monitor daemon tailstatd.
protected void	writeDagFileTail(DagInfo dinfo) It writes the relations making up the DAG in the dag file and and closes the file handle to it.
protected void	writeDOTFile(String filename, ADag dag) Writes out the DOT file in the submit directory.
protected String	writeOutBraindump(File directory, ADag workflow, String dax, String dagFile) Writes out the braindump.txt file for a partition in the partition submit directory.

Methods inherited from class org.griphyn.cPlanner.code.generator.Abstract

getFileBaseName, getWriter

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mSeparator

public static final String mSeparator

The nice separator, define once, use often.

See Also:

Constant Field Values

mLogger

protected LogManager mLogger

The LogManager object which is used to log all the messages.

mTCHandle

protected TransformationCatalog mTCHandle

Handle to the Transformation Catalog.

mPoolHandle

private PoolInfoProvider mPoolHandle

Handle to the pool provider.

mPoolClass

protected String mPoolClass

Specifies the implementing class for the pool interface. Contains the name of the class that implements the pool interface the user has asked at runtime.

mProjectMap

private Map mProjectMap

Maps pool handles to project names. This is needed for some remote scheduling systems (not limited to LSF) for accounting purposes.

mQueueMap

private Map mQueueMap

Maps pool handles to queue names. This is needed for some remote scheduling systems (not limited to LSF) for accounting purposes.

mWalltimeMap

private Map mWalltimeMap

Maps pool handles to walltimes for that pool.

mDagWriter

protected PrintWriter mDagWriter

The file handle to the .dag file. A part of the dag file is printed as we write the submit files, to insert the appropriate postscripts for handling exit codes.

mTempLogFile

protected String mTempLogFile

The name of the log file in the /tmp directory

mDone

protected boolean mDone

A boolean indicating whether the files have been generated or not.

mConcreteWorkflow

protected ADag mConcreteWorkflow

The workflow for which the code has to be generated.

mStyleFactory

protected CondorStyleFactory mStyleFactory

Handle to the Style factory, that is used for this workflow.

mGridStartFactory

protected GridStartFactory mGridStartFactory

The handle to the GridStart Factory.

mInitializeGridStart

protected boolean mInitializeGridStart

A boolean indicating whether grid start has been initialized or not.

Constructor Detail

CondorGenerator

public CondorGenerator()

The default constructor.

Method Detail

initialize

public void initialize(PegasusBag bag)
                throws CodeGeneratorException

Initializes the Code Generator implementation. Initializes the various writers.

Specified by:

initialize in interface CodeGenerator

Overrides:

initialize in class Abstract

Parameters:

bag - the bag of initialization objects.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

generateCode

public void generateCode(ADag dag)
                  throws CodeGeneratorException

Generates the code for the concrete workflow in Condor DAGMAN and CondorG input format.

Parameters:

dag - the concrete workflow.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

generateCode

public void generateCode(ADag dag,
                         SubInfo job)
                  throws CodeGeneratorException

Generates the code (condor submit file) for a single job.

Parameters:

dag - the dag of which the job is a part of.

job - the SubInfo object holding the information about that particular job.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

startMonitoring

public boolean startMonitoring()
                        throws VTorInUseException

Starts monitoring of the workflow by invoking a workflow monitor daemon tailstatd. The tailstatd is picked up from the default path of $PEGASUS_HOME/bin/tailstatd.

Specified by:

startMonitoring in interface CodeGenerator

Overrides:

startMonitoring in class Abstract

Returns:

boolean indicating whether could successfully start the monitor daemon or not.

Throws:

VTorInUseException - in case the method is called before the submit files have been generated.

reset

public void reset()
           throws CodeGeneratorException

Resets the Code Generator implementation.

Specified by:

reset in interface CodeGenerator

Overrides:

reset in class Abstract

Throws:

CodeGeneratorException - in case of any error occuring code generation.

constructMap

private Map constructMap(String propValue)

Constructs a map with the numbers/values to be passed in the RSL handle for certain pools. The user ends up specifying these through the properties file. The value of the property is of the form poolname1=value,poolname2=value....

Parameters:

propValue - the value of the property got from the properties file.

Returns:

Map

initializeDagFileWriter

protected void initializeDagFileWriter(String filename,
                                       DagInfo dinfo)
                                throws CodeGeneratorException

Initializes the file handler to the dag file and writes the header to it.

Parameters:

filename - basename of dag file to be written.

dinfo - object containing daginfo of type DagInfo .

Throws:

CodeGeneratorException - in case of any error occuring code generation.

writeDOTFile

protected void writeDOTFile(String filename,
                            ADag dag)
                     throws CodeGeneratorException

Writes out the DOT file in the submit directory.

Parameters:

filename - basename of dot file to be written .

dag - the ADag object.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

writeDagFileTail

protected void writeDagFileTail(DagInfo dinfo)
                         throws CodeGeneratorException

It writes the relations making up the DAG in the dag file and and closes the file handle to it.

Parameters:

dinfo - object containing daginfo of type DagInfo.

Throws:

CodeGeneratorException

printDagString

protected void printDagString(String str)
                       throws CodeGeneratorException

Writes a string to the dag file. When calling this function the file handle to file is already initialized.

Parameters:

str - The String to be printed to the dag file.

Throws:

CodeGeneratorException

getLogFileName

protected String getLogFileName()

The name of the condor log file that is used for the dag that is being written into the condor formal. This name is used as the default logfile for all the jobs unless it is over ridden in the condor namespace. It is the full path to the log file stored in the local temporary directory.

Returns:

the name of the log file.

getLogFileSymlinkName

protected String getLogFileSymlinkName(ADag dag)

Returns the path to the symlink that is created to the original log file in the temporary directory.

Parameters:

dag - the concrete workflow.

Returns:

the path to symlink.

writeOutBraindump

protected String writeOutBraindump(File directory,
                                   ADag workflow,
                                   String dax,
                                   String dagFile)
                            throws IOException

Writes out the braindump.txt file for a partition in the partition submit directory. The braindump.txt file is used for passing to the tailstatd daemon that monitors the state of execution of the workflow.

Parameters:

directory - the directory in which the braindump file needs to be written to.

workflow - the concerte workflow.

dax - the corresponding DAX file containing the abstract workflow.

dagFile - the basename of the .dag file that is written out while generating output.

Returns:

the absolute path to the braindump file.txt written in the directory.

Throws:

IOException - in case of error while writing out file.

generateLogFileSymlink

protected boolean generateLogFileSymlink(String logFile,
                                         String symlink)

This method generates a symlink to the actual log file written in the local temp directory. The symlink is placed in the dag directory.

Parameters:

logFile - the full path to the log file.

symlink - the full path to the symlink.

Returns:

boolean indicating if creation of symlink was successful or not

getDAGFilename

protected String getDAGFilename(ADag dag,
                                String suffix)

Returns the name of the file on the basis of the metadata associated with the DAG. In case of Condor dagman, it is the name of the .dag file that is written out. The basename of the .dag file is dependant on whether the basename prefix has been specified at runtime or not by command line options.

Parameters:

dag - the dag for which the .dag file has to be created.

suffix - the suffix to be applied at the end.

Returns:

the name of the dagfile.

getDAGMANOutFilename

protected String getDAGMANOutFilename(ADag dag)

Returns the basename of the file, that contains the output of the dagman while running the dag generated for the workflow. The basename of the .out file is dependant on whether the basename prefix has been specified at runtime or not by command line options.

Parameters:

dag - the DAG containing the concrete workflow

Returns:

the name of the dagfile.

concreteDagEmpty

protected void concreteDagEmpty(String filename,
                                ADag dag)
                         throws CodeGeneratorException

A callout method that dictates what needs to be done in case the concrete plan that is generated is empty. It just logs a message saying the plan is empty.

Parameters:

filename - Filename of the dag to be written of type String.

dag - the concrete dag that is empty.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

handleCondorVarForJob

protected void handleCondorVarForJob(SubInfo sinfo)
                              throws CodeGeneratorException

It updates/adds the condor variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally. In case of clashes of Condor variables from various sources the following order is followed,property file, pool config file and then dax.

Parameters:

sinfo - The SubInfo object containing the information about the job.

Throws:

CodeGeneratorException

handleTransferOfExecutable

protected void handleTransferOfExecutable(SubInfo sinfo)
                                   throws CodeGeneratorException

It changes the paths to the executable depending on whether we want to transfer the executable or not. If the transfer_executable is set to true, then the executable needs to be shipped from the submit host meaning the local pool. This function changes the path of the executable to the one on the local pool, so that it can be shipped.

Parameters:

sinfo - the SubInfo containing the job description.

Throws:

CodeGeneratorException

applyStyle

protected void applyStyle(SubInfo job,
                          PrintWriter writer)
                   throws CodeGeneratorException

Applies a submit file style to the job, according to the fact whether the job has to be submitted directly to condor or to a remote jobmanager via CondorG and GRAM. If no style is associated with the job, then for the job running on local site, condor style is applied. For a job running on non local sites, globus style is applied if none is associated with the job.

Parameters:

job - the job on which the style needs to be applied.

writer - the PrintWriter stream to the submit file for the job.

Throws:

CodeGeneratorException - in case of any error occuring code generation.

handleEnvVarForJob

protected void handleEnvVarForJob(SubInfo sinfo)

It updates/adds the environment variables that are got through the Dax with the values specified in the properties file, pool config file or adds some variables internally. In case of clashes of environment variables from various sources the following order is followed,property file, transformation catalog, pool config file and then dax. At present values are not picked from the properties file.

Parameters:

sinfo - The SubInfo object containing the information about the job.

handleGlobusRSLForJob

protected void handleGlobusRSLForJob(SubInfo sinfo)

It updates/adds the the Globus RSL parameters got through the dax that are in SubInfo object. In addition inserts the additional rsl attributes that can be specified in the properties file or the pool config files in the profiles tags. In case of clashes of RSL attributes from various sources the following order is followed,property file, pool config file and then dax.

Parameters:

sinfo - The SubInfo object containing the information about the job.

gridstart

private String gridstart(PrintWriter writer,
                         SubInfo job,
                         boolean isGlobusJob)
                  throws CodeGeneratorException

This function creates the stdio handling with and without gridstart. Please note that gridstart will become the default by end 2003, and no gridstart support will be phased out.

Parameters:

writer - is an open stream for the Condor submit file.

job - is the job information structure.

isGlobusJob - is true, if the job generated a line universe = globus, and thus runs remotely. Set to false, if the job runs on the submit host in any way.

Returns:

A possibly empty string which contains things that need to be added to the "globusrsl" clause. The return value is only of interest for isGlobusJob==true calls.

Throws:

CodeGeneratorException - in case of any error occuring code generation.