Welcome to Pegasus WMS’s documentation!

User Guide

• 1. Introduction
  • 1.1. What are scientific workflows?
  • 1.2. How to Convert Existing Applications Into a Workflow
• 2. Deployment Scenarios
  • 2.1. Hosted Pegasus Instances
  • 2.2. HTCondor Pools
  • 2.3. HPC Clusters - System Install
  • 2.4. HPC Clusters - User Install
  • 2.5. HPC Clusters - Specific Systems
  • 2.6. Remote HPC Clusters
  • 2.7. Cloud (AWS, Google, JetStream, …)
  • 2.8. Localhost
• 3. Installation
  • 3.1. Prerequisites
  • 3.2. Optional Software
  • 3.3. Environment
  • 3.4. RHEL / CentOS / Rocky / Alma / SL
  • 3.5. Ubuntu
  • 3.6. Debian
  • 3.7. Mac OS X
  • 3.8. Pegasus from Tarballs
  • 3.9. Pegasus Python Packages for PyPi
  • 3.10. Mixing Environments (system/venv/conda/…)
• 4. Tutorial
• 5. Example Workflows
  • 5.1. PegasusHub
  • 5.2. Pegasus Init
  • 5.3. Pegasus Init Execution Environments
• 6. Creating Workflows
  • 6.1. Abstract Workflows
  • 6.2. Catalogs
  • 6.3. Best Practices For Developing Portable Code
• 7. Running Workflows
  • 7.1. Executable Workflows (DAG)
  • 7.2. Data Staging Configuration
  • 7.3. Pegasus-Plan
  • 7.4. Basic Properties
• 8. Submit Directory Details
  • 8.1. Layout
  • 8.2. HTCondor DAGMan File
  • 8.3. Sample Condor DAG File
  • 8.4. Kickstart Record
  • 8.5. Reading a Kickstart Output File
  • 8.6. Jobstate.Log File
  • 8.7. Pegasus Workflow Job States and Delays
  • 8.8. Braindump File
  • 8.9. Pegasus static.bp File
• 9. Monitoring, Debugging and Statistics
  • 9.1. Workflow Status
  • 9.2. Plotting and Statistics
  • 9.3. Dashboard
  • 9.4. Hooks
• 10. Containers
  • 10.1. Overview
  • 10.2. Singularity vs Apptainer
  • 10.3. Configuring Workflows To Use Containers
  • 10.4. Containers on OSG
  • 10.5. Container Execution Model
  • 10.6. Staging of Application Containers
• 11. Data Transfers
  • 11.1. Data Staging Configuration
  • 11.2. Local versus Remote Transfers
  • 11.3. Controlling Transfer Parallelism
  • 11.4. Symlinking Against Input Data
  • 11.5. Addition of Separate Data Movement Nodes to Executable Workflow
  • 11.6. Staging of Executables
  • 11.7. Staging of Worker Package
  • 11.8. Staging of Worker Package into Containers
  • 11.9. Staging of Job Checkpoint Files
  • 11.10. Bypass Input File Staging
• 12. Optimizing Workflows for Efficiency and Scalability
  • 12.1. Optimizing Short Jobs / Scheduling Delays
  • 12.2. Job Clustering
  • 12.3. Hierarchical Workflows
  • 12.4. Optimizing Data Transfers
  • 12.5. Job Throttling
  • 12.6. Increase Memory Requirements for Retries
  • 12.7. Slot Partitioning and CPU Affinity in Condor
• 13. Migration Notes
  • 13.1. Migrating From Pegasus 4.9.X to Pegasus 5.0
  • 13.2. Moving From DAX3 to Pegasus.api
  • 13.3. Migrating From Pegasus 4.5.X to Pegasus 4.9.x
  • 13.4. Migrating From Pegasus <4.5 to Pegasus 4.5.X
  • 13.5. Migrating From Pegasus 3.1 to Pegasus 4.X
  • 13.6. Migrating From Pegasus 2.X to Pegasus 3.X
• 14. Glossary

Reference Guide

• 1. Workflow API
  • 1.1. Python
  • 1.2. Java
  • 1.3. R
  • 1.4. Monitoring REST API
• 2. Configuration
  • 2.1. Differences between Profiles and Properties
  • 2.2. Profiles
  • 2.3. Properties
• 3. Catalogs
  • 3.1. Data Discovery (Replica Catalog)
  • 3.2. Resource Discovery (Site Catalog)
  • 3.3. Executable Discovery (Transformation Catalog)
• 4. Mapping Refinement Steps
  • 4.1. Data Reuse
  • 4.2. Site Selection
  • 4.3. Job Clustering
  • 4.4. Addition of Data Transfer and Registration Nodes
  • 4.5. Addition of Create Dir and Cleanup Jobs
  • 4.6. Code Generation
• 5. Data Management
  • 5.1. Replica Selection
  • 5.2. Supported Transfer Protocols
  • 5.3. Credentials Management
  • 5.4. Staging Mappers
  • 5.5. Output Mappers
  • 5.6. Data Cleanup
  • 5.7. Metadata
  • 5.8. Integrity Checking
• 6. Monitoring
  • 6.1. Monitoring Database
  • 6.2. Stampede Workflow Events
  • 6.3. Publishing to AMQP Message Servers
• 7. PegasusLite
  • 7.1. Setting the directory on the local filesystem for the job
  • 7.2. Setting the environment in PegasusLite for your job
  • 7.3. Specify Compute Job in PegasusLite to run on different node
• 8. Pegasus Service
  • 8.1. Service Administration
  • 8.2. Dashboard
  • 8.3. Running Pegasus Service under Apache HTTPD
  • 8.4. Ensemble Manager
• 9. CWL Support
• 10. Variable Expansion
• 11. Command Line Tools
  • 11.1. pegasus-analyzer
  • 11.2. pegasus-aws-batch
  • 11.3. pegasus-cluster
  • 11.4. pegasus-config
  • 11.5. pegasus-configure-glite
  • 11.6. pegasus-cwl-converter
  • 11.7. pegasus-dagman
  • 11.8. pegasus-db-admin
  • 11.9. pegasus-em
  • 11.10. pegasus-exitcode
  • 11.11. pegasus-globus-online-init
  • 11.12. pegasus-globus-online
  • 11.13. pegasus-graphviz
  • 11.14. pegasus-halt
  • 11.15. pegasus-init
  • 11.16. pegasus-integrity
  • 11.17. pegasus-keg
  • 11.18. pegasus-kickstart
  • 11.19. pegasus-metadata
  • 11.20. pegasus-monitord
  • 11.21. pegasus-mpi-cluster
  • 11.22. pegasus-mpi-keg
  • 11.23. pegasus-plan
  • 11.24. pegasus-rc-client
  • 11.25. pegasus-rc-converter
  • 11.26. pegasus-remove
  • 11.27. pegasus-run
  • 11.28. pegasus-s3
  • 11.29. pegasus-sc-converter
  • 11.30. pegasus-service
  • 11.31. pegasus-statistics
  • 11.32. pegasus-status
  • 11.33. pegasus-submitdir
  • 11.34. pegasus-tc-converter
  • 11.35. pegasus-transfer
  • 11.36. pegasus-version
• 12. Funding, citing, and anonymous usage statistics
  • 12.1. Citing Pegasus in Academic Works
  • 12.2. Usage Statistics Collection

Pegasus Development

• Code Documentation
  • 1.1.1. Pegasus.api package
  • Java
• Schemas
  • DAX XML Schema

Indices and tables

• Index

• Module Index

• Search Page