Useful Nextflow logs

Questions

  • How can we use the workflow logs generated by Nextflow to understand the resource needs of our data?
  • What were the main observations and possible limitations when running the pipeline?


Nextflow has some inbuilt reporting features that are used by the nf-core/rnaseq pipeline. The nf-core/rnaseq pipeline outputs these reports to the directory we ran the pipeline from (/home/training/base_directory/working_directory). The information presented in these reports helps to get an estimate of the required resources for independant processes and assign them for future runs.

Remember back to the run command we used? We specified two flags to create these reports:

Workflow execution report

-with-report excecution_report.html

This instructs Nextflow to generate an execution report as an .html file. This file is broken down into:

  • Summary: execution status, run command, execution time, other workflow metadata.
  • Resources: plots of disturbution of CPU, RAM usage for each workflow process.
  • Tasks: all tasks executed by the run command, their status, and command run.

Workflow timeline report

-with-timeline timeline_report.html

This instructs Nextflow to generate a timeline report as an .html file. This file contains bar plots for each process, how long they took to run and the maximum amount of memory consumed.

Key points

  • Nextflow provides easy to understand html log files, which can be used to understand the system resource requirements.

All materials copyright Sydney Informatics Hub, University of Sydney