Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
 PREV CLASS   NEXT CLASS FRAMES    NO FRAMES    
SUMMARY: NESTED | FIELD | CONSTR | METHOD DETAIL: FIELD | CONSTR | METHOD

edu.isi.pegasus.planner.dax
Class ADAG

java.lang.Object
  extended by edu.isi.pegasus.planner.dax.ADAG

public class ADAG
extends Object

 This class provides the Java API to create DAX files.

 The DAX XML SCHEMA is available at http://pegasus.isi.edu/schema/dax-3.3.xsd
 and documentation available at http://pegasus.isi.edu/wms/docs/schemas/dax-3.3/dax-3.3.html

 The DAX consists of 6 parts the first 4 are optional and the last is optional.
  1. file:Used as "In DAX" Replica Catalog (Optional)

  2. executable: Used as "In DAX" Transformation Catalog (Optional)

  3. transformation: Used to describe compound executables. i.e. Executable depending on other executables (Optional)

  4. job|dax|dag: Used to describe a single job or sub dax or sub dax. Atleast 1 required.

  5. child: The dependency section to describe dependencies between job|dax|dag elements. (Optional)

 To generate an example DIAMOND DAX run the ADAG Class as shown below
 java ADAG filename
 NOTE: This is an illustrative example only. Please see examples directory for a working example

 Shown below are some of the steps in creating a  DIAMOND DAX.
  1. Create a new ADAG object

    ADAG dax = new ADAG("test");

  2. Add notifications to the workflow

    j3.addNotification(WHEN.start,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");

    j3.addNotification(WHEN.at_end,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");

  3. Create a File object
    You only need to add entries to this section if you want to use an "IN-DAX" Replica Catalog"

    File fa = new File("f.a");

    1. Add MetaData entry to the file objects

      fa.addMetaData("string", "foo", "bar");
      fa.addMetaData("int", "num", "1");

    2. Add Profile entry to the file objects

      fa.addProfile("env", "FOO", "/usr/bar");
      fa.addProfile("globus", "walltime", "40");

    3. Add PFN to the File object

      fa.addPhysicalFile("file:///scratch/f.a", "local");

    4. Add the File object to the Replica Catalog section of the DAX

      dax.addFile(fa);

  4. Create an Executable object
    You only need to add entries to this section if you want to use an "IN-DAX" Replica Catalog"

    Executable preprocess = new Executable("pegasus", "preproces", "1.0");
    1. Set the Executable.ARCH and Executable.OS for the executable. Default is x86 and LINUX

      preprocess.setArchitecture(Executable.ARCH.x86).setOS(Executable.OS.LINUX);

    2. Set the executable as available to be staged. Default is installed executable

      preprocess.unsetInstalled();

    3. Add the physical location PFN of the executable. In case of stageable executables the path should be a url

      preprocess.addPhysicalFile(new PFN("file:///opt/pegasus/default/bin/keg"));

    4. Add Profile and MetaData objects to the executable

      preprocess.addProfile(Profile.NAMESPACE.globus, "walltime", "120");
      preprocess.addMetaData("string", "project", "pegasus");

    5. Add the Executable object to the ADAG object

      dax.addExecutable(preprocess);

  5. Create a Transformation object : compound Executable (Executable depending on other executable and files)

    Transformation diamond = new Transformation("pegasus", "diamond", "1.0");
    1. Add the sub executable for this transformation

      diamond.uses(preprocess).uses(findrange).uses(analyze);

    2. Add the sub files(e.g config files) for this transformation

      diamond.uses(new File("config", File.LINK.INPUT));

    3. Finally Add the Transformation to the ADAG object

      dax.addTransformation(diamond);

  6. Create a Job object

    Job j1 = new Job("j1", "pegasus", "preprocess", "1.0", "j1");
    1. Add Arguments to the job object

      j1.addArgument("-a","preprocess")
      j1.addArgument("-T","60").addArgument("-i",fa);
      j1.addArgument("-o").addArgument(fb1).addArgument(fb2);

    2. Add the Files that are used by this job

      j1.uses(fa, File.LINK.INPUT);
      j1.uses(fb1, File.LINK.OUTPUT);
      j1.uses(new File("f.b2"), File.LINK.OUTPUT);

    3. Add the Notifications to this job

      j3.addNotification(WHEN.start,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");
      j3.addNotification(WHEN.at_end,"/usr/local/pegasus/libexec/notification/email -t notify@example.com -f workflow@example.com");

    4. Add Profiles to the job

      j1.addProfile(Profile.NAMESPACE.dagman, "pre", "20");

    5. Add the Job object to ADAG

      dax.addJob(j1);

  7. Add a DAG object

    DAG j2 = new DAG("j2", "findrange.dag", "j2");
    j2.uses(new File("f.b1"), File.LINK.INPUT);
    j2.uses(new File("f.c1"), File.LINK.OUTPUT);
    j2.addProfile(Profile.NAMESPACE.dagman, "pre", "20");
    j2.addProfile("condor", "universe", "vanilla");
    dax.addDAG(j2);

  8. Add a DAX job object.

    DAX j3 = new DAX("j3", "findrange.dax", "j3");
    j3.addArgument("--site").addArgument("local");
    j3.uses(new File("f.b2"), File.LINK.INPUT);
    j3.uses(new File("f.c2"), File.LINK.OUTPUT);
    j3.addProfile("ENV", "HAHA", "YADAYADAYADA");
    dax.addDAX(j3);

  9. Add the Job dependencies

    Dependencies can be added by specifiying the job id's like so

    dax.addDependency("j1", "j2", "1-2").addDependency("j1", "j3", "1-3");

    or by specifying the job|dax|dag objects directly as below

    dax.addDependency(j1,j3);

  10. Finally write the dax to a file

    dax.writeToFile("diamond.dax");

Version:
$Revision: 4507 $
Author:
Gaurang Mehta gmehta at isi dot edu

Field Summary
private  int mCount
          The Count of the number of dax objects : N
private  Map<String,List<Parent>> mDependencies
          Map of Dependencies between Job,DAX,DAG objects.
private  Set<Executable> mExecutables
          The list of Executable objects
private  List<File> mFiles
          The list of edu.isi.pegasus.planner.dax.File objects
private  int mIndex
          The Index of the dax object.
private  List<Invoke> mInvokes
          List of Notification objects
private  Map<String,AbstractJob> mJobs
          The List of Job,DAX and DAG objects
private  LogManager mLogger
           
private  String mName
          The Name / Label of the DAX
private  Set<Transformation> mTransformations
          The List of Transformation objects
private  XMLWriter mWriter
          Handle the XML writer
static String SCHEMA_LOCATION
          The "not-so-official" location URL of the DAX schema definition.
static String SCHEMA_NAMESPACE
          The "official" namespace URI of the site catalog schema.
static String SCHEMA_NAMESPACE_XSI
          XSI SCHEMA NAMESPACE
static String SCHEMA_VERSION
          The version to report.
 
Constructor Summary
ADAG(String name)
          The Simple constructor for the DAX object
ADAG(String name, int index, int count)
          DAX Constructor
 
Method Summary
private  ADAG addAbstractJob(AbstractJob ajob)
          Add AbstractJob to the DAX
private  ADAG addAbstractJobs(List<AbstractJob> ajobs)
          Add AbstractJobs to the DAX
 ADAG addDAG(DAG dag)
          Add a DAG job to the DAX
 ADAG addDAGs(List<DAG> dags)
          Add multiple DAG jobs to the DAX
 ADAG addDAX(DAX dax)
          Add a DAX job to the DAX
 ADAG addDAXs(List<DAX> daxs)
          Add multiple DAX jobs to the DAX
 ADAG addDependency(AbstractJob parent, AbstractJob child)
          Add a parent child dependency between two jobs,dax,dag
 ADAG addDependency(AbstractJob parent, AbstractJob child, String label)
          Add a parent child dependency with a dependency label
 ADAG addDependency(String parent, String child)
          Add a parent child dependency between two jobs,dax,dag
 ADAG addDependency(String parent, String child, String label)
          Add a parent child dependency with a dependency label
 ADAG addExecutable(Executable executable)
          Add Executable to the DAX
 ADAG addExecutables(List<Executable> executables)
          Add Multiple Executable objects to the DAX
 ADAG addFile(File file)
          Add a RC File object to the top of the DAX.
 ADAG addFiles(List<File> files)
          Add Files to the RC Section on top of the DAX
 ADAG addInvoke(Invoke.WHEN when, String what)
          Add a Notification for this Workflow
 ADAG addInvoke(Invoke invoke)
          Add a Notification for this Workflow
 ADAG addInvokes(List<Invoke> invokes)
          Add a List of Notifications for this Workflow
 ADAG addJob(Job job)
          Add Job to the DAX
 ADAG addJobs(List<Job> jobs)
          Add multiple Jobs to the DAX
 ADAG addNotification(Invoke.WHEN when, String what)
          Add a Notification for this Workflow
 ADAG addNotification(Invoke invoke)
          Add a Notification for this Workflow
 ADAG addNotifications(List<Invoke> invokes)
          Add a List of Notifications for this Workflow
 ADAG addTransformation(Transformation transformation)
          Add Transformation to the DAX
 ADAG addTransformations(List<Transformation> transformations)
          Add Multiple Transformation to the DAX
private  boolean containsAbstractJob(AbstractJob ajob)
          Check if an abstractjob exists in the DAX
private  boolean containsAbstractJobId(String ajobid)
          Check if a jobid exists in the DAX
 boolean containsDAG(DAG dag)
          Check if a DAG job exists in the DAX
 boolean containsDAGId(String dagid)
          Check if a DAG job id exists in the DAX
 boolean containsDAX(DAX dax)
          Check if a DAX job exists in the DAX
 boolean containsDAXId(String daxid)
          Check if a DAX job id exists in the DAX
 boolean containsExecutable(Executable executable)
          Checks if a given executable exists in the DAX based Transformation Catalog
 boolean containsJob(Job job)
          Check if a job exists in the DAX
 boolean containsJobId(String jobid)
          Check if a jobid exists in the DAX
 boolean containsTransformation(Transformation transformation)
          Checks if a given Transformation exists in the DAX based Transformation Catalog
private static ADAG Diamond()
           
private  AbstractJob getAbstractJob(String ajobid)
          Returns an abstract Job with id ajobid if present otherwise null.
 DAG getDAG(String dagid)
          Returns a DAG object with id dagid if present otherwise null.
 DAX getDAX(String daxid)
          Returns a DAX object with id daxid if present otherwise null.
 Job getJob(String jobid)
          Returns a Job object with id jobid if present otherwise null.
static void main(String[] args)
          Create an example DIAMOND DAX
 void toXML(XMLWriter writer)
          Generates a DAX representation.
 void writeToFile(String daxfile)
          Generate a DAX File out of this object;
 void writeToSTDOUT()
          Generate a DAX representation on STDOUT.
 void writeToWriter(Writer writer, boolean close)
          Generate a DAX representation and pipe it into the Writer
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

SCHEMA_NAMESPACE

public static final String SCHEMA_NAMESPACE
The "official" namespace URI of the site catalog schema.

See Also:
Constant Field Values

SCHEMA_NAMESPACE_XSI

public static final String SCHEMA_NAMESPACE_XSI
XSI SCHEMA NAMESPACE

See Also:
Constant Field Values

SCHEMA_LOCATION

public static final String SCHEMA_LOCATION
The "not-so-official" location URL of the DAX schema definition.

See Also:
Constant Field Values

SCHEMA_VERSION

public static final String SCHEMA_VERSION
The version to report.

See Also:
Constant Field Values

mName

private String mName
The Name / Label of the DAX

mIndex

private int mIndex
The Index of the dax object. I out of N

mCount

private int mCount
The Count of the number of dax objects : N

mJobs

private Map<String,AbstractJob> mJobs
The List of Job,DAX and DAG objects

See Also:
DAG, DAX, Job, AbstractJob

mTransformations

private Set<Transformation> mTransformations
The List of Transformation objects

See Also:
Transformation

mExecutables

private Set<Executable> mExecutables
The list of Executable objects

See Also:
Executable

mFiles

private List<File> mFiles
The list of edu.isi.pegasus.planner.dax.File objects

See Also:
File

mDependencies

private Map<String,List<Parent>> mDependencies
Map of Dependencies between Job,DAX,DAG objects. Map key is a string that holds the child element reference, the value is a List of Parent objects

See Also:
Parent

mInvokes

private List<Invoke> mInvokes
List of Notification objects

mWriter

private XMLWriter mWriter
Handle the XML writer

mLogger

private LogManager mLogger
Constructor Detail

ADAG

public ADAG(String name)
The Simple constructor for the DAX object

Parameters:
name - DAX LABEL

ADAG

public ADAG(String name,
            int index,
            int count)
DAX Constructor

Parameters:
name - DAX Label
index - Index of DAX out of N DAX's
count - Number of DAXS in a group
Method Detail

addInvoke

public ADAG addInvoke(Invoke.WHEN when,
                      String what)
Add a Notification for this Workflow

Parameters:
when -
what -
Returns:
ADAG

addNotification

public ADAG addNotification(Invoke.WHEN when,
                            String what)
Add a Notification for this Workflow

Parameters:
when -
what -
Returns:
ADAG

addInvoke

public ADAG addInvoke(Invoke invoke)
Add a Notification for this Workflow

Parameters:
invoke -
Returns:
ADAG

addNotification

public ADAG addNotification(Invoke invoke)
Add a Notification for this Workflow

Parameters:
invoke -
Returns:
ADAG

addInvokes

public ADAG addInvokes(List<Invoke> invokes)
Add a List of Notifications for this Workflow

Parameters:
invokes -
Returns:
ADAG

addNotifications

public ADAG addNotifications(List<Invoke> invokes)
Add a List of Notifications for this Workflow

Parameters:
invokes -
Returns:
ADAG

addFile

public ADAG addFile(File file)
Add a RC File object to the top of the DAX.

Parameters:
file - File object to be added to the RC section
Returns:
ADAG
See Also:
File

addFiles

public ADAG addFiles(List<File> files)
Add Files to the RC Section on top of the DAX

Parameters:
files - List List of file objects to be added to the RC Section
Returns:
ADAG
See Also:
File

addExecutable

public ADAG addExecutable(Executable executable)
Add Executable to the DAX

Parameters:
executable - Executable to be added
Returns:
ADAG
See Also:
Executable

addExecutables

public ADAG addExecutables(List<Executable> executables)
Add Multiple Executable objects to the DAX

Parameters:
executables - List of Executable objects to be added
Returns:
ADAG
See Also:
Executable

containsExecutable

public boolean containsExecutable(Executable executable)
Checks if a given executable exists in the DAX based Transformation Catalog

Parameters:
executable -
Returns:
boolean

addTransformation

public ADAG addTransformation(Transformation transformation)
Add Transformation to the DAX

Parameters:
transformation - Transformation object to be added
Returns:
ADAG
See Also:
Transformation

addTransformations

public ADAG addTransformations(List<Transformation> transformations)
Add Multiple Transformation to the DAX

Parameters:
transformations - List of Transformation objects
Returns:
ADAG
See Also:
Transformation

containsTransformation

public boolean containsTransformation(Transformation transformation)
Checks if a given Transformation exists in the DAX based Transformation Catalog

Parameters:
transformation - Transformation
Returns:
boolean

addAbstractJob

private ADAG addAbstractJob(AbstractJob ajob)
Add AbstractJob to the DAX

Parameters:
ajob - AbstractJob
Returns:
ADAG
See Also:
Job, DAG, DAX, AbstractJob

addAbstractJobs

private ADAG addAbstractJobs(List<AbstractJob> ajobs)
Add AbstractJobs to the DAX

Parameters:
ajobs - AbstractJob
Returns:
ADAG
See Also:
Job, DAG, DAX, AbstractJob

getAbstractJob

private AbstractJob getAbstractJob(String ajobid)
Returns an abstract Job with id ajobid if present otherwise null.

Parameters:
ajobid -
Returns:

containsAbstractJob

private boolean containsAbstractJob(AbstractJob ajob)
Check if an abstractjob exists in the DAX

Parameters:
ajob -
Returns:

containsAbstractJobId

private boolean containsAbstractJobId(String ajobid)
Check if a jobid exists in the DAX

Parameters:
ajobid -
Returns:

addJob

public ADAG addJob(Job job)
Add Job to the DAX

Parameters:
job -
Returns:
ADAG
See Also:
Job, AbstractJob

addJobs

public ADAG addJobs(List<Job> jobs)
Add multiple Jobs to the DAX

Parameters:
jobs -
Returns:
ADAG
See Also:
Job, AbstractJob

containsJob

public boolean containsJob(Job job)
Check if a job exists in the DAX

Parameters:
job -
Returns:

containsJobId

public boolean containsJobId(String jobid)
Check if a jobid exists in the DAX

Parameters:
jobid -
Returns:

getJob

public Job getJob(String jobid)
Returns a Job object with id jobid if present otherwise null.

Parameters:
jobid -
Returns:

getDAX

public DAX getDAX(String daxid)
Returns a DAX object with id daxid if present otherwise null.

Parameters:
daxid -
Returns:

getDAG

public DAG getDAG(String dagid)
Returns a DAG object with id dagid if present otherwise null.

Parameters:
dagid -
Returns:

addDAG

public ADAG addDAG(DAG dag)
Add a DAG job to the DAX

Parameters:
dag - the DAG to be added
Returns:
ADAG
See Also:
DAG, AbstractJob

addDAGs

public ADAG addDAGs(List<DAG> dags)
Add multiple DAG jobs to the DAX

Parameters:
dags - List of DAG jobs to be added
Returns:
ADAG
See Also:
DAG, AbstractJob

containsDAG

public boolean containsDAG(DAG dag)
Check if a DAG job exists in the DAX

Parameters:
dag -
Returns:

containsDAGId

public boolean containsDAGId(String dagid)
Check if a DAG job id exists in the DAX

Parameters:
dagid -
Returns:

addDAX

public ADAG addDAX(DAX dax)
Add a DAX job to the DAX

Parameters:
dax - DAX to be added
Returns:
ADAG
See Also:
DAX, AbstractJob

addDAXs

public ADAG addDAXs(List<DAX> daxs)
Add multiple DAX jobs to the DAX

Parameters:
daxs - LIST of DAX jobs to be added
Returns:
ADAG
See Also:
DAX, AbstractJob

containsDAX

public boolean containsDAX(DAX dax)
Check if a DAX job exists in the DAX

Parameters:
dax -
Returns:

containsDAXId

public boolean containsDAXId(String daxid)
Check if a DAX job id exists in the DAX

Parameters:
daxid -
Returns:

addDependency

public ADAG addDependency(String parent,
                          String child)
Add a parent child dependency between two jobs,dax,dag

Parameters:
parent - String job,dax,dag id
child - String job,dax,dag,id
Returns:
ADAG

addDependency

public ADAG addDependency(AbstractJob parent,
                          AbstractJob child)
Add a parent child dependency between two jobs,dax,dag

Parameters:
parent - Job|DAX|DAG object
child - Job|DAX|DAG object
Returns:

addDependency

public ADAG addDependency(String parent,
                          String child,
                          String label)
Add a parent child dependency with a dependency label

Parameters:
parent - String job,dax,dag id
child - String job,dax,dag id
label - String dependency label
Returns:
ADAG

addDependency

public ADAG addDependency(AbstractJob parent,
                          AbstractJob child,
                          String label)
Add a parent child dependency with a dependency label

Parameters:
parent - Job|DAX|DAG object
child - Job|DAX|DAG object
label - String label for annotation
Returns:
ADAG

writeToFile

public void writeToFile(String daxfile)
Generate a DAX File out of this object;

Parameters:
daxfile - The file to write the DAX to

writeToSTDOUT

public void writeToSTDOUT()
Generate a DAX representation on STDOUT.

writeToWriter

public void writeToWriter(Writer writer,
                          boolean close)
Generate a DAX representation and pipe it into the Writer

Parameters:
writer - A Writer object
close - Whether writer should be closed on return.

toXML

public void toXML(XMLWriter writer)
Generates a DAX representation.

Parameters:
writer -

main

public static void main(String[] args)
Create an example DIAMOND DAX

Parameters:
args -

Diamond

private static ADAG Diamond()
Overview  Package   Class  Use  Tree  Deprecated  Index  Help 
 PREV CLASS   NEXT CLASS FRAMES    NO FRAMES    
SUMMARY: NESTED | FIELD | CONSTR | METHOD DETAIL: FIELD | CONSTR | METHOD
Copyright © 2011 The University of Southern California. All Rights Reserved.