Introduction and agenda

Overview

Teaching: 5 min
Exercises: 0 min

Questions

• SIH introduction and workshop agenda

Objectives

• Introduce SIH

• Workshop overview

The Sydney Informatics Hub

The Sydney Informatics Hub (SIH) is a Core Research Facility of the University of Sydney. Core Research Facilities centralise essential research equipment and services that would otherwise be too expensive or impractical for individual Faculties to purchase and maintain. The classic example might be the room-size electron-microscopes, built into specialised rooms in the Sydney Microscopy & Microanalysis unit.

Artemis HPC is the main piece of equipment supported by SIH. However, we also provide a wide range of research services to aid investigators, such as:

• Training
• Statistics consulting
• Modeling/Simulation/Visualisation consulting and platforms
• Bioinformatics consulting and platforms
• Research Data Management consulting and platforms
• Data Science & Research Engineering project work

We also aim to cultivate a data community at USyd, organising monthly Hacky Hours, outside training events (eg NVIDIA, Pawsey Center), conferences, and data/coding-related events.

NCI and USyd/SIH partnership

The University of Sydney is part of a collaboration agreement with NCI.

HPC resources at NCI are available to USyd researchers through two formal arrangements:

1. LIEF grant LE190100021 (8 universities, including USyd, obtain NCI resources through this grant)
2. USyd-Intersect collaboration agreement

At present, these resource pools are managed as a combined scheme through Intersect and SIH.

This means that your use of NCI Gadi is supported by both NCI and USyd, and as such you have two possible sources of help. For job and system related questions, contact NCI. For questions about USyd allocations, contact SIH.

About this workshop

One of our key priorities is facilitating researcher’s access to the appropriate resource. For many groups, Artemis HPC is not able to meet the computing needs of your group. At SIH, we can assist you in gaining access to other national HPC facilities, including:

• NCI Gadi
• UQ Flashlite
• Pawsey Supercomputing Centre
• Commercial cloud

Today, we will be specifically looking at NCI Gadi, and briefly covering the topics outline below. This is an ‘onboarding’ or ‘quickstart’ course, designed for users who already have an existing NCI user ID and project. Others are welcome to follow along, but please note that assumed knowledge is general knowledge of HPC systems, HPC job submission and Linux command line ability.

Key Gadi-specific commands will be demonstrated during the presentation of this workshop. There are also practice jobs at the end of the workshop, designed to demonstrate multi-node, massively parallel jobs and how to optimise and execute these on NCI Gadi. Participants with their own NCI user ID and project code are encouraged to test the commands and submit the practice jobs (sample data will be provided).

Agenda

• About NCI Gadi
  • Access
  • Useful resources and where to get help
  • The system: Gadi queues, hardware and filesystem
• Accounting
  • Service units and charge rates
  • Checking compute resource availability
  • Checking disk resource availability
• PBS jobs and Gadi-specific commands
  • Key differences between Gadi and Artemis directives and commands
• Data transfer
  • Use of Gadi copyq and Artemis dtq
• Software
  • Global versus local
  • Making locally installed software compatible with module commands
• Short break
  • The remaining sections are optional, but recommended
• Optimisation
  • Understanding efficiency and scalability
  • Benchmarking
  • Selecting optimal resources and queues
• Example parallel job
  • Run a multi-node parallel job with OpenMPI and nci-parallel
  • Examples included for bioinformatics and simulation: session will be run with two breakut groups

Key Points

• SIH facilitates access to compute resources including NCI Gadi

• This workshop is a Gadi quickstart