Gadi overview

This section will point you to the right sections of the NCI documentation and user guides to get you started on the Gadi HPC system.

Resources

Gadi technical summary

NCI Gadi is one of Australia’s most powerful supercomputers, designed to support advanced computational research.

Component Details
Compute - Nodes: 4,962
- Processors: Intel Sapphire Rapids, Cascade Lake, Skylake, and Broadwell CPUs
- GPUs: NVIDIA V100 and DGX A100 GPUs
- Performance: Over 10 petaflops of peak performance :contentReferenceoaicite:0
Storage - Disk Drives: 7,200 4-Terabyte hard disks in 120 NetApp disk arrays
- Capacity: 20 Petabytes total usable capacity
- Performance: 980 Gigabytes per second maximum performance :contentReferenceoaicite:1
Filesystems - Total Capacity: Approximately 90 Petabytes
- Global Lustre Filesystems: Five, with an aggregate I/O performance of around 450 GB/second
- IO Intensive Platform: Dedicated filesystem using 576 2-Terabyte NVMe drives, achieving around 960 Gigabytes per second cumulative performance :contentReferenceoaicite:2
Archival Storage - Capacity: Over 70 Petabytes of archival project data stored in state-of-the-art magnetic tape libraries :contentReferenceoaicite:3
Networking - Interconnect: 100-gigabit network links connecting high-performance computing with high-performance data :contentReferenceoaicite:4
Cloud Systems - Nirin Cloud: High-availability and high-capacity zone integrated with Gadi and NCI’s multi-Petabyte national research data collections, comprising Intel Broadwell and Sandy Bridge processors and NVIDIA K80 GPUs :contentReferenceoaicite:5

Conditions of use

All users of NCI agree that they will keep themselves informed of, and comply with, all relevant legislation and The Australian National University policies and rules.

All users must acknowledge and understand that a breach of these will result in not only a loss of access to NCI resources but the user may be subject to Federal criminal prosecution resulting in fines and/or gaol legislated under the Acts listed on the conditions of use and policies page.

Login nodes

These nodes are the gateway for Gadi for users to access the resources of the HPC cluster. It is how you log in to Gadi, move around the filesystem, submit jobs to the scheduler, and do small tasks like view the contents of a file.

When you first ssh into Gadi, you are working on a login node. These are distinct from the compute nodes and are not designed for running compute tasks or transferring data.

While you can run simple commands on the login nodes on Gadi, commands are throttled. This means they execute more slowly than on the compute nodes and are automatically killed if they exceed CPU, memory and time restrictions that are in place. This is to ensure the login nodes are not over-loaded and remain responsive for all users. As such, all complex or resource-intensive tasks must be submitted to the job scheduler for execution on the cluster nodes.

Compute nodes

These nodes are the workhorses of any HPC. They are dedicated for executing computational tasks as delegated by the job scheduler when you submit a job. There are various types of compute nodes with different hardware, built for different purposes on Gadi. Depending on the resource requirements of your job (e.g. high memory, GPUs) and the queue you specify, your job will be sent to a specific type of compute node.

Please see the queue structure on Gadi and the queue limits pages for detailed information on the compute node specifications.

For details of how to choose and request a specific queue for your job, please visit the section on PBS scripts and job submission.

Data mover nodes

These nodes are designed specifically for fast data movement. You can use these nodes to transfer files to and from Gadi at high-speed.

Please visit the Gadi file transfer page for more details.

A comprehensive guide to data transfer between Gadi and University of Sydney RDS, including example commands, template scripts and recommendations, can be found in the section on transferring data.

Filesystems

$HOME

When you first log in to Gadi, you’ll be placed in your personal $HOME directory (i.e. /home/555/aa1234). You are the only person who can access this directory. No work should be done in here due to the strict 10 GB storage limit, but you may wish to install things like custom R or Python libraries here. It is backed up.

You can navigate back here at any point if required by running:

cd ~

/scratch

All Gadi projects have a dedicated /scratch allocation that is only accessible to members of your project. This is only intended for active work and not for long-term storage. This is not backed up and any files not accessed for 100 days will be purged from the system, so be sure to back up your work to RDS following our data transfer guide.

Visit the NCI scratch file management page for details on the purge policy and how to recover files from ‘quarantine’.

Scratch will be /scratch/<nci-project-id>. Each member of a project will have read/write permissions for this parent directory, as well as their /scratch/<nci-project-id>/<nci-user-id> directory.

You can navigate to your /scratch space by running:

cd /scratch/<project>

/g/data

Some Gadi projects have a dedicated /g/data allocation that is only accessible to members of that group. This in intended for long-term large data storage, for example large reference files or databases that are regularly required for your compute jobs. Compute node jobs can read /g/data so this is an ideal place to store those files so that they are not subject to /scratch purge. Files that are on /g/data for this kind of use should also have a copy on RDS, since /g/data/ is not backed up.

If you are unsure if your project has a /g/data allocation, you can check by running:

cd /g/data/<project>

Scratch is provided at no additional cost to users, however g/data has a cost per TB involved. This cost is covered by the NCI-Sydney Scheme for projects that have a justified need of it. To request /g/data storage space, please request this via an email to nci-sydney.scheme@sydney.edu.au with a brief justification.

/apps

This directory is accessible to all Gadi users. It is a read-only system containing centrally installed software applications and their module files.

Please visit the software section for more information on global apps and alternatives.