Intermediate Artemis HPC: Automating analysis with job arrays

Synopsis: This course is aimed at researchers who currently use the University of Sydney’s High Performance Computing (HPC) cluster ‘Artemis’ and would like to learn how to automate multiple-run analyses with job arrays, and do simple bash scripting on Artemis.

The live training consists of instruction and practical exercises, covering: Simple job arrays using the subjob-index as parameter; using the job-name string as parameter; and using configuration for more complex analyses.

Target audience: Students and staff who are already using the Artemis HPC and who would like to learn how to use job arrays and bash scripting to automate the submission of multiple jobs simultaneously.

Follow-on courses: This course is part three of the Artemis HPC Training Series.

It follows the introductory courses Introduction to Artemis HPC and Data transfer and RDS for HPC.

The final lesson, Matlab on Artemis: The MDCS, is an optional course for users of MATLAB who would like to submit jobs directly to the cluster from their local computers.


You are assumed to have your own Artemis account, know how to connect to Artemis, navigate its filesystem, and submit jobs to the PBS Scheduler. If you are not familiar with Artemis, you must take Introduction to Artemis HPC before attempting this course. This will teach you the minimum requirements for you to be able to attempt the Intermediate course.

You must have competency on the Unix/Linux command line. If you are interested in learning HPC but have no Unix/Linux command-line skills, you MUST first take an Introduction to Unix/Linux course, followed by Introduction to Artemis HPC.


Course survey!

Please fill out our course survey before you leave!

Help us help you! :smiley:


Setup Install software and data needed to begin the lesson
00:00 1. Welcome back to Artemis HPC Who are the Sydney Informatics Hub?
00:15 2. Automating with Array Jobs How to run multiple similar jobs
00:45 3. Break! Break
00:50 4. Automating with Arrays Jobs Using array jobs to ‘batch’ analyses
01:20 5. Break! Break
01:25 6. More ways to automate Further ways to automate jobs, with arrays and scripts
02:05 Finish

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.