About NCI Gadi
Overview
Teaching: 5 min
Exercises: 4 minQuestions
Software and workflows suitable for Gadi
Objectives
Gadi’s hardware, queues, filesystem
Access
Gadi hostname
The system
National Computational Infrastructure (NCI) is a services facility that provides high performance computing (HPC), cloud and data services to Australian researchers.
Gadi is the name of NCI’s HPC facility, located at ANU (Canberra). The system is suitable for:
- software that can utilise multiple nodes
- highly CPU efficient workflows
- scalable workflows
- users with moderate HPC experience
Gadi placed 27th in November 2020 in the TOP500’s rankings for the most powerful supercomputers globally. Gadi is currently the largest supercomputer in the Southern Hemisphere, but it will be superseded by Pawsey’s new supercomputer (PACER program) in 2021.
The hardware
The table below summarises the compute power and hardware on some of Australia’s HPCs, and how they compare to 2020’s largest supercomputer (1st in the TOP500’s rankings, November 2020), the Japanese Fugaku.
| HPC | USyd Artemis | NCI Gadi | Pawsey PACER | UQ FlashLite | Japanese Fugaku |
|---|---|---|---|---|---|
| FLOPS | NA | 9.26 PFLOPS | 50 PFLOPS | 65 TFLOPS | 415 PFLOPS |
| Cores | 7,636 | 204,032 | 200,000+ | 1,632 (ScaleMP vSMP) | 7,299,072 |
| GPU | 108 NVIDIA V100 | 640 NVIDIA V100 | 750+ AMD GPUs | 0 | 0 |
| High memory nodes | 3 @ 6 TB | 50 @ 1.5 TB, 10 @ 1 TB, 3 @ 3 TB | >548 TB total | 68 @ 512 GB (ScaleMP vSMP) | 158,976 @ 32 GB |
| Job Scheduler | PBSPro | PBSPro | NA | PBSPro | Fugaku’s scheduler (pjsub, pjstat, pjdel) |
| Networking | 56 Gbps FDR Infiniband | 200 Gbps HDR InfiniBand, Dragonfly + | 400 Gbps CISCO spine-leaf | 56Gbps Infiniband spine-leaf | Tofu interconnect D |
| Storage | 495 TB /scratch, 407 TB /project | 20 PB high speed, 47 PB for large data, 50 PB tape | 2.7 PB high speed, 126 PB tape storage | 150+ TB high speed, 326 TB NVMe SSD | 150 PB shared storage, 1.6 TB NVMe SSD |
Queues
Look at the Gadi queue limits page (you might want to bookmark this!)
Note that the walltime limit is 48 hours, or less if requesting a larger number of cores or using specialised queues.
You might also want to bookmark the Gadi queue structure page.
Note that there are specialised queues with specialised hardware. Selecting the right queue for your jobs is critical to ensuring efficient use of the system.
Filesystems
| Filesystem | Description |
|---|---|
| /home | Backed up, 10 GiB fixed quota per user. |
| /scratch | Not backed up, temporary files, auto purge policy applied. |
| /apps | Read-only, centrally installed software applications and their module files. |
| $PBS_JOBFS | Not backed up, local to the node, I/O intensive data. |
| /g/data | Not backed up, long-term large data files. |
| massdata | Backed up, archiving large data files. |
Access
Access to compute time on NCI Gadi is restricted and competitive. Access can be granted by:
All projects are created through my.nci.org.au
Logging in
Access to Gadi is via command line:
ssh <user>@gadi.nci.org.au
If you have an account on NCI Gadi, please log in now with your preferred terminal client
Useful resources and where to get help
Key Points
Gadi is a much larger HPC than Artemis
Access is merit-based or paid for
Log in to Gadi via ssh to
gadi.nci.org.au