Part 4 Maximizing the utilization of GPU resources on-premise and in the cloud

Part 4 Maximizing the utilization of GPU resources on-premise and in the cloud

• 1.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. SCHEDULING OF GPUS ON PREMISE AND IN CLOUD WITH GRID ENGINE Bill Bryce, VP Products Univa
• 2.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. Agenda • Challenges of using GPUs with Grid Engine • Applications & GPUs • Meta-data for GPUs in Grid Engine • GPU & CPU binding • Environment & Containment of GPUs • Accounting in Grid Engine for GPUs • Putting it all together & Demo • Upcoming improvements in Grid Engine for GPUs.
• 3.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. Challenges of using GPUs with Grid Engine Application o Many applications work ‘out of the box’ but some are difficult to manage Metrics o in the form of resources available to schedule Meta-data o provide hints to the scheduler so it has information to make a decision Scheduling policy o scheduler needs to make a ‘good decision’ based on available data Topology o part of the scheduling that needs to be considered – sometimes improves performance Environment o GPU jobs require information to run correctly such as ‘which GPU’ was chosen. Containment o When a GPU job runs it needs to run contained on the selected resources Accounting o GPU usage is different than CPUs.
• 4.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. Grid Engine & GPUs: Applications 4 Most applications can be run by simply ‘submitting’ to Grid Engine: qsub –l “cloud=1,gpu=1(A100)” –pe smp 8 run-tensorflow-gpu.sh However With many frameworks and applications you need: o Application packaging o Virtualization o Containerization Many sites use one, or more: o Modules o Python virtual environments o Containers: Docker & Singularity Virtualization is common when using Applications with GPUs
• 5.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. Grid Engine & GPUs: Containers 5 Packaging the application in containers simplifies its use with Grid Engine and GPUs Docker o A tool that packages an application, filesystem, and all other dependencies into an easily distributable software package. The resulting images are used to create containers. o Docker has a daemon – makes some things difficult for Grid Engine o https://www.docker.io Singularity o A tool that packages an application and all other dependencies into an image that can be used to create user space untrusted (or trusted ) containers o Singularity does not have a daemon. It runs in user space. o Singularity can use Docker images o https://sylabs.io Both tools are typically used in HPC and Grid Engine clusters.
• 6.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. Grid Engine & GPUs: Container Images 6 Once a container system is chosen you need images that contain applications You can create your own images, but easier to use existing Docker Hub https://hub.docker.com Singularity Cloud https://cloud.sylabs.io/home However Docker Hub and Singularity Cloud do not specialize in GPU container images: NVIDIA NGC A catalog of GPU optimized software for HPC, AI/ML and Deep Learning. https://ngc.nvidia.com
• 7.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. Meta-Data: RSMAPs for GPUs in Grid Engine GPUs must be modelled in Grid Engine so the scheduler can understand what to do with them; this is not automatic. RSMAPs • A special host consumable complex • Allows to configure strings for naming each unit • Maps resource units (strings) to jobs, manages them like consumables RSMAPs allow for users submit Grid Engine jobs that request one or more GPU resources. qconf -sc #name shortcut type relop requestable consumable default urgency aapre #--------------------------------------------------------------------------- GPUTYPE GPUTYPE STRING == YES NO NONE 0 NO GPU GPU RSMAP <= YES YES NONE 0 NO
• 8.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. Meta-Data: RSMAPs for GPUs in Grid Engine Initialize the RSMAP: complex_values gpu=2(gpu0[cuda_id=0,device=/dev/nvidia0] gpu1[cuda_id=1,device=/dev/nvidia1]) Submit job: qsub -l gpu=1 ... Show granted device with: qstat –j granted devices host: /dev/nvidia0 The job can discover the assigned IDs by reading the: SGE_HGR_<complexname> environment variable Optionally the job can use CUDA_VISIBLE_DEVICES by using SET_CUDA_VISIBLE_DEVICES=TRUE Access to CUDA Devices can also be restricted via . This also works for Docker jobs
• 9.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. Topology: GPU and CPU Binding GPU Topology o GPUs can be associated with CPUs that are near to them complex_values GPU=2(GPU0:SccCC GPU1:SCCcc) GPU Binding o In conjunction with modern cgroups support GPUs can be bound to a particular job cgroup_params gpu_devices=true,devices=/dev/nvidia[0-7]
• 10.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. Environment and Containment of GPUs Grid Engine Jobs that require GPUs and containers need to run on Hardware with: • one or more GPUs • Correct NVIDIA drivers installed • docker installed • nvidia-docker2 subsystem installed • Optionally NVIDIA DCGM installed When a job runs it needs information from UGE to determine what GPU was chosen: • The job can discover the assigned IDs using the SGE_HGR_<complexname> environment variable • The prefix HGR stands for Hard Granted Resource • With cgroups binding there is no need to use SGE_HGR_<complexname> Containment is achieved using: o Docker containers that map the GPUs into the job qsub –l docker=true,docker_images=”*nvidia/cuda*”,gpu_map=2 -xd “—runtime=nvidia” job.sh
• 11.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. Accounting on GPUs Univa Grid Engine 8.6.7 and later can optionally collect GPU job metrics • Metrics are enabled on each host with an execd parameter: execd_param ENABLE_DCGM_JOB_USAGE=TRUE Grid Engine cluster configuration parameter gpu_job_usage controls what DCGM parameters are displayed: • For example: gpu_job_usage all gpu_job_usage energyConsumed, maxGpuMemoryUsed, overallHealth
• 12.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. Putting it all together to run on premise or in Cloud 12 Okay, let’s recap the steps…. 1. Choose an application, for example Tensorflow 2. Decide what container technology to use: Docker or other 3. Choose Hardware to run the job – GPU in preferred Cloud 4. Optionally choose an image with the NVIDIA drivers and Docker or 5. Ensure the NVIDIA drivers are installed 6. Ensure Docker and nvidia-docker2 are installed 7. Configure Docker for nvidia ngc https://ngc.io 8. Configure Grid Engine with RSMAPs for GPUs and add a cloud complex. Packaging and running your application this way makes it portable and it can run on premise and in Cloud with minimal changes – for Grid Engine all that is needed is -l cloud
• 13.
  © Altair Engineering,Inc. Proprietary and Confidential. All rights reserved. • Coming Soon in less than a month, 8.6.16: • Support for A100 and MIG, DCGM2.0 is in development with the following features: • Grid Engine supports the Docker --gpus flag. • This allows MIGs to be selected for Grid Engine jobs running in Docker containers: qrsh -l docker,docker_images="*cuda*",docker_api_version=1.4 -xd "--gpus device="0:0"" -xd "-- runtime=nvidia" • The —gpus and device=”0:0”” select GPU id 0 and MIG id 0 • MIGs are not directly requestable from qrsh or qsub but RSMAPs can be configured to recognize MIGs • New DOCKER_RESOLVE_CUDA_ID=TRUE in qmaster params enables this functionality • The result is you request a RSMAP and that assigns the GPU and MIG qrsh -l docker,docker_images="*cuda*",docker_api_version=1.4 -l gpu=1 -xd "--gpus device="${gpu(0)}"" -xd "--runtime=nvidia” • For 8.6.7 (Q1 2021) • Grid Engine Execd will automatically detect MIGs on a machine with A100 cards Upcoming Improvements to Grid Engine 13
• 14.
  THANK YOU altair.com #ONLYFORWARD