Data transfer and Research Data Storage with High Performance Computing

Synopsis: This course is aimed at researchers who want to learn how to transfer data between your local computer, an external source, the Research Data Store (RDS) and the Artemis High Performance Computer (HPC). This course will introduce smart strategies for managing your data and how to back up HPC output onto the RDS. The skills and tools learned in this session can be applicable to any remote storage and HPC environments.

The live training consists of instruction and practical exercises, covering tools such as: scp, rsync, mv, cp, filezilla, winscp, ssh, smbclient.

Target audience: Students and staff who are would like to use the Research Data Store and the Artemis HPC.

Follow-on courses: This course is part two of the Artemis HPC Training Series. It follows on from the introductory courses Introduction to Artemis HPC.

Recorded Lesson: You can watch the recorded lesson from April 24, 2020, here: https://cloudstor.aarnet.edu.au/plus/s/Yb3c4nqE48PwoT4

Prerequisites

You should have competency on the Unix/Linux command line. If you are interested in learning HPC but have no Unix/Linux command-line skills, you should first take an Introduction to Unix/Linux course, followed by Introduction to Artemis HPC. These are scheduled regularly on campus.

OWN LAPTOP REQUIRED.


Course survey!

Please fill out our course survey before you leave!

Help us help you! :smiley:

Schedule

Setup Install software and data needed to begin the lesson
00:00 1. Data transfer and RDS for HPC What is the Reasearch Data Store?
00:25 2. Transfer from LOCAL to Artemis HPC What tools can you use to copy data to a remote machine?
00:45 3. Transfer from WEB to Artemis HPC What should you think about when transferring large datasets?
01:05 4. Transfer between remote RDS and Artemis HPC
01:35 5. Mounting RDS on your system How can you access RDS from your local machine?
01:45 6. Summary and other Data Storage options What other Data Storage options are availble?
01:50 7. BONUS: transfer with SMB to the Classic RDS How can you access classic RDS from your local machine?
How can you access classic RDS from Artemis?
02:10 Finish

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.