Many new technologies and new paradigms are emerging in the research community to support science workflows, including the use of Data Transfer Nodes (DTNs), distributed compute and data infrastructure, research testbeds, and new inter-domain federated orchestrators. Workflow management systems (WMS), like Pegasus, are evolving to be more resource aware in order to make more intelligent decisions on dispatching and coordinating computation and data transfer jobs.

For global-scale scientific applications, Open Exchange Points (OXP) [1] are the critical communication infrastructure to connect the compute resources and the data intensive instruments that are often located in different continents [2]. For nearly a decade,  Software Defined Networking (SDN) [3] has been introduced into the OXP domains to create a Software Defined eXchange (SDX) [4] that can dramatically improve the network utilization, programmability, and the efficiency to manage network resources. This opens the door to allow the resource orchestrator and application software to make the best use of the networking capabilities in real time.

The collaboration between the Pegasus and AtlanticWave-SDX (AW-SDX) projects demonstrates our vision to develop an end-to-end resource orchestration framework that will enable intelligent interactions between the scientific workflows and the network substrate.

The AtlanticWave-SDX (AW-SDX) project (NSF OAC-1451024 and 1451024, 2015-2025) changed how domain scientists and network engineers can use Open Exchange Points (OXPs) to support science workflows. The AW-SDX team developed an inter-domain SDX platform to link a large number of data sources and scientific users to the advanced distributed OXPs across Miami, Atlanta, Latin America, and South Africa towards an open, innovative platform while continuously enhancing their production capacity and service capabilities.

AtlanticWave-SDX

Challenges in Wide-Area Networks (WAN) are being addressed by the application of network virtualization and network programmability solutions using SDN and Network Functions Virtualization (NFV) [5] technologies. Network virtualization and programmable networks are two key enablers that facilitate agile, fast, and more economical network infrastructures, as well as service development, deployment, and provisioning. Typically, SDNs that cross multiple domains are constructed manually, involving significant coordination and effort by network operators. OXPs serve as intersections for connecting and facilitating the exchange of data between different Research and Education domain networks. They are critical cyberinfrastructure in the interconnectivity of data over long geographic distances, switching data flows from one research and education network to the next, to its destination. They require this unique effort in networking science to perform efficiently and effectively. Operationalizing the transit of data flows across OXPs is increasingly important to minimize the impact from events on network services (hardware failures and soft failures). End-to-end network paths for these data flows are not under the control of any individual organization. Multiple R&E network operators collaborate and must coordinate to establish geographically distributed end-to-end paths – an effort that can take days to weeks.

AW-SDX serves as a distributed experimental SDX to support research, experimental deployments, prototyping, and interoperability testing on national and international scales. AW-SDX is a response to the growing demand to support end-to-end services spanning multiple SDN domains. AtlanticWave-SDX is comprised of two components: (1) a network infrastructure development component to bridge 100G of network capacity between R&E backbone networks in the U.S. and S. America, as shown in Figure 1; and (2) an innovation component to build a distributed intercontinental experimental SDX between the U.S. and S. America, by leveraging open exchange point resources at SoX (Atlanta), AMPATH (Miami), and Southern Light (São Paulo, Brazil), as shown in Figure 2.

Publication:  

Contacts: julio@fiu.edu