sc20.hpcworkshops.comAWS HPC Workshops :: AWS HPC Workshops

AWS-Logo_White-Color SC20 Tutorial Resources Tutorial Agenda Updates Accessing Your Lab Account Lab Prep - Before Starting Prerequisites a. Sign in to the Console b. Create a Cloud9 Environment c. Opt - Work with the AWS CLI d. Opt - Create an S3 Bucket e. Opt - Launch the EC2 Dashboard f. Opt - Create an EC2 Instance g. Opt - Create an IAM Role Summary Lab I ⁃ Create an HPC Cluster a. Install AWS ParallelCluster b. Configure ParallelCluster c. Create a Cluster Config d. Build an HPC Cluster e. Log in to Your Cluster f. Submit your first HPC job g. Summary Lab II ⁃ Serverless Computing a. Services & Architecture b. Define a new IAM policy - Create the S3 Bucket - Build an IAM Policy with Cloudformation c. Apply the policy to your cluster d. Create the serverless function e. Add code to your function f. Configure the function - Adding environment variables - Configure the function timeout g. Attach an IAM Role to Lambda - Create the lambda IAM policy - Attach the IAM policy to the role h. Bind Lambda with API Gateway - Create the API Gateway and REST Resource - Bind the REST API to your Lambda function - Deploy your new API i. Interact with Slurm through your API - Set the API parameters - List instances and partitions - Submitting jobs through cURL j. Summary & cleanup Lab III ⁃ Storage Lab a. Install AWS ParallelCluster b. Create your HPC Cluster c. Examine the File System d. About Lazy File Loading d. Install IOR Benchmark Tool e. Test IO Performance f. View Metrics with CloudWatch g. Summary and Cleanup Lab IV ⁃ Performance Monitoring Grafana Dashboards a. Cluster configuration b. Build your HPC cluster c. Login d. Review the grafana dashboards e. Interactive dashboard monitoring g. Compute nodes monitoring f. HPCG & IOR metrics g. Summary Troubleshooting Optional - Amazon SNS Notification a. Update cluster with SNS Policy b. Setup SNS topic c. Publish to SNS topic d. Slurm Job Notification using SNS e. Summary & Cleanup Opt I ⁃ Simulations on AWS Batch a. Lab Environment b. Workshop Initial Setup c. Build Your AMI with Packer d. Create a Docker Repository e. Set up Compute Environment f. Set up a Job Queue g. Set up a Job Definition h. Describe Your Environment i. Run a Single Job j. Run an Array Job k. Next Steps & Clean up Opt II ⁃ Remote Visualization DCV using ParallelCluster a. Create a cluster configured with NICE DCV b. Build an HPC Cluster with NICE DCV c. Connect to your NICE DCV Session e. Terminate Your Cluster DCV using web browser/native client a. Deploy EC2 instance with NICE DCV b. Connect to NICE DCV EC2 Instance c. Connect to Remote Desktop Session d. Terminate Your Instance Opt III ⁃ Low Latency Network a. EFA Basics b. Create an HPC Cluster with EFA c. Examine an EFA enabled instance d. Work With Intel MPI e. Download, compile and run the OSU Benchmark f. Delete Your EFA Cluster Opt IV ⁃ Distributed ML a. Upload training data to S3 b. Create a distributed ML cluster c. Run single node data preprocessing with Slurm d. Run PyTorch Data Parallel training on ParallelCluster e. Delete Distributed ML Cluster More Github AWS - HPC Authors Feedback / Questions? Tags Clear History Star Fork Privacy | Site Terms | © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. navigation SC20 Tutorial Welcome to the Best Practices for HPC in the Cloud tutorial at SC 20 that will be delivered on November 9th 2020 and November 11th 2020. We are excited to virtually meet you, and look forward to answering your questions! The website will be maintained after SC20, labs will also be available on https://www.hpcworkshops.com within a few weeks after the conference. We are always looking at ways to make this tutorial even better. Please provide feedback and comments here: https://bit.ly/sc20cloud . Sandbox were available between Nov 9th to 10th for the duration of the tutorial. If you would like to run through the labs at a later stage on your own, with your company or institution, please contact us at sc20tutorial@amazon.com so we can follow-up with you. Important Information & Updates For the tutorial recordings delivered on the APJ timezone , please use Gitter to for questions or support during the tutorial Need help? Email : you can contact us directly before, during and after the conference through this email: sc20tutorial@amazon.com Gitter : a Gitter room can be access for the duration of the tutorial which you can use to contact us for advice or help: https://gitter.im/aws-sc20/general Tutorial Content & Resources Tutorial Resources Before and during the tutorial you may be interested in going through the following sections: Agenda of the tutorial. FAQ answers to common questions will be communicated here during the tutorial. Lab account on how to access your lab account. Presentations Slides The last version of the tutorial slides are available in your Tutorial attendee packet. Do not hesitate to contact us at sc20tutorial@amazon.com . SC20 Hands-on Labs Throughout the tutorial we will be going through the following labs: Lab 0: Environment Prep : It has to be done before running the first lab. It will grant you access to a web-based development environment and terminals. This ensure that every one can run the labs regardless of their operating system. Lab 1: Create an HPC Cluster : You will be lead to create your first HPC system in the Cloud. Lab 2: Serverless Computing : We will see how to build an API interface and submit jobs on Slurm using a serverless (Lambda) function. Lab 3: Storage Lab : In this lab you will learn how to build a Lustre file system in the Cloud, no need to hold a PhD in storage for that. Lab 4: Performance Monitoring at Scale : This lab gives an overview of different methods and tools that can be used to assess application and infrastructure performance of your HPC cluster in the cloud. Optional Labs If you are interested to run additional labs don’t hesitate to run one of these: Optional 1: Running Serverless Simulations : You will be introduced to AWS Batch which provides scheduling and orchestration functionalities to run jobs at scale in the cloud. Optional 2: Remote Visualization : In this part we will introduce you to NICE DCV in order to visualize data residing in the cloud. Optional 3: Low Latency Network : This part drives your through Elastic Fabric Adapter (EFA), a low latency network interface which can be used to run your tightly coupled workloads in the cloud. You may want to run the benchmark GPCNET as a stretch goal to collect the tail latency on EFA. This lab cannot be tested during SC20 with your Event Engine Account due to the limited resources that these accounts provide Optional 4: Distributed Machine Learning : This lab will enable you to run a Machine Learning distributed workload using the PyTorch DistributedDataParallel API. Accessing Your Lab Account Once you get your credentials during the tutorial, access your account through https://dashboard.eventengine.run . Report an issue on GitHub Contact AWS HPC Team Learn...