Download to read offline
Uploaded byMatthew Vaughn
Jetstream: Adding Cloud-based Computing to the National Cyberinfrastructure
Jetstream is a cloud-based computing resource funded by the NSF, designed to democratize access to shared cyberinfrastructure for researchers across diverse scientific domains. It enables on-demand computational resources with customizable environments, aims to support researchers with varying computational needs, and provides access to cloud-native technologies. The system is currently in early operations, allowing users with active XSEDE accounts to utilize its services for research and education.
More Related Content
Similar to Jetstream: Adding Cloud-based Computing to the National Cyberinfrastructure
Jetstream: Adding Cloud-based Computing to the National Cyberinfrastructure
- 1. funded by theNational Science Foundation Award #ACI-1445604 Jetstream: Adding Cloud-based Computing to the National Cyberinfrastructure Matthew Vaughn(@mattdotvaughn) ORCID 0000-0002-1384-4283 Director, Life Science Computing co-PI, Jetstream | Cyverse | Araport Texas Advanced Computing Center
- 2. Funded by theNational Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ What is Jetstream? • A resource to expand the community of users who benefit from NSF investment in shared cyberinfrastructure • Production cloud system supporting all domains of science and engineering research sponsored by the NSF • Provide on-demand interactive computing and analysis • Enable configurable environments and architectures • Support computational reproducibility and sharing • Democratizes access to cloud-native technology and software • Focuses on ease of use, but also on maintaining flexiblility
- 3. Expanding NSF XD’sreach and impact Around 299,000 researchers, educators, & learners received NSF support in 2012-2013 – Only 1.5% completed a computation, data analysis, or visualization task on XD program resources – Less than 3% had an XSEDE Portal account – 70% of researchers surveyed* claimed to be resource constrained Why aren’t they using XD systems? – Activation energy is pretty high – HPC resources are scarce and not well-matched to their needs – They just don’t need that much capability * https://www.xsede.org/xsede-nsf-release-cloud-survey-report
- 4. Funded by theNational Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Expanding NSF XD’s reach and impact Capability class machines Traditional HPC, HTC systems
- 5. Funded by theNational Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Who do we expect will use Jetstream? • Researchers, developers, and scientists who… – Require somewhere between 4 and a few hundred cores RIGHT NOW and for the foreseeable future, but not forever – Want the ability to fully customize the OS and configuration of their computing setup – Wish to move cloud-native workflows to/from an academic environment – Need interactive mode access to a computing and analysis system
- 6. Funded by theNational Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Who do we expect will use Jetstream? • Also… – Science gateway operators using Jetstream as either the frontend or processor for scientific jobs – Anyone who is evaluating or experimenting with software not traditionally supported on XD systems – STEM Educators teaching on a variety of subjects
- 7. Funded by theNational Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Diverse research domains • Biology: iPlant and Galaxy VMs, enabling access to and use of new analytical codes in various modalities • Earth Science: VMs capable of requesting NSIDC data and running common routines to enable more effective research and better analyses of data • Field Station Research: VM-based data collection and analysis tools to support data sharing and collaboration • GIS: Deliver the CyberGIS toolkit and provide access to ArcGIS in a VM using IU’s existing site license • Network Science: Build VMs with CIShell tool builders to deliver network analysis tools interactively • Social Sciences: Create VMs that allow selection of data from the Odum Institute in a way that retains provenance and version information
- 8. Funded by theNational Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ 21st century workforce development • Specialized virtual Linux desktops and applications to enable research and research education at small colleges and universities – HBCUs (Historically Black Colleges and Universities) – MSIs (Minority Serving Institutions) – Tribal colleges – Higher-education institutions in EPSCoR States • Also, complete democratization of access to cloud-native technologies and approaches
- 9. Platform Overview Atmosphere APIGlobusAuth Atmo Services XSEDE Accounting OpenStack CEPH Indiana University OpenStack CEPH TACC OpenStack CEPH Potentially, Others Web App
- 10. Funded by theNational Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Systems Overview
- 11. Flavor vCPUs RAMStorage Per Node m.tiny 1 2 20 46 m.small 2 4 40 23 m.medium 6 16 130 7 m.large 10 30 230 4 m.xlarge 22 60 460 2 m.xxlarge 44 120 920 1 VM Host Configuration • Dual Intel E-2680v3 “Haswell” • 24 physical cores/node @ 2.5 GHz (Hyperthreading on) • 128 GB RAM • Dual 1 TB local disks • 10GB dual uplink NIC • Running KVM Hypervisor Hardware Specifics CEPH Storage • 20x Dell 730xd per cloud • 2x10Gbs bonded NIC per 730xd • Running CEPH 0.94.5 Hammer • Configured as OpenStack Storage • Storage is XSEDE-allocated • Implemented on backend as OpenStack Volumes • Each user gets 10 volumes up to 500GB total storage • Exploring object storage as well but that’s in the future
- 12. Integration with XSEDEvia Globus Auth Atmosphere Web App uses and Globus Auth implements industry- standard Oauth2 • Leaves us flexibility on identity and access • Globus Auth implements (in beta) password grant Oauth flow, which means Jetstream access can be entirely scripted
- 22. Programmatic Access Web ServiceAPIs • Openstack - Official and unofficial clients + libs (i.e. boto) • EC2* – Integration with AWS-specific code • Atmosphere - Very beta. Getting language libraries “soon” • Preview @ http://docs.atmospherev2.apiary.io/ *Still quite finicky Automation, Orchestration, and Workflow • Marathon/MESOS • https://www.youtube.com/v/VzZfwHLmcL0 • Docker Machine* + Swarm • Apache Airavata • CloudMan & Elasticluster* Configuration Management Tools • Vagrant & Terraform (Hashicorp) • Chef
- 23. Example: Marathon, MESOS,Unidata
- 24. Funded by theNational Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ What comes next? • Both cloud systems + all software components installed, configured, and operational • “Early operations mode” is underway • End of March 2016: 38 XSEDE projects and 250+ users • Acceptance review scheduled with NSF in early May • Full, unrestricted operations after system is accepted • Soliciting Research allocation requests NOW plus Startup and Education allocations
- 25. Allow any userwith active XSEDE User Portal account to use a small (but functional) slice of Jetstream • Sign up for XUP Account • Sign into User Portal • Click “Trial Jetstream Access” button • Have (restricted) access to Jetstream in ~30 minutes or less Analogous to the free tier that makes it so easy to get started on the public cloud The “Easy Button”
- 26. Funded by theNational Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Partners Construction Application / Community LeadsManagement & Operations Vendors
- 27.
- 28. Funded by theNational Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ How can I use Jetstream? • An XSEDE User Portal (XUP) account is required. They are free! Get one at https://portal.xsede.org • Read the Allocations Overview - https://portal.xsede.org/allocations- overview • Write a successful allocation request – start with a Startup or Education request - https://portal.xsede.org/successful-requests
- 29. Funded by theNational Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Where can I get help or learn more? • Production: – User guides: https://portal.xsede.org/user-guides – XSEDE KB: https://portal.xsede.org/knowledge-base – Email: help@xsede.org – Campus Champions: https://www.xsede.org/campus-champions – Training Videos / Virtual Workshops (TBD) • Early use: – http://jetstream-cloud.org/ – Early use: jethelp@iu.edu