Migrating to 24.04

Nextflow 24.04 was released on May 20, 2024.

New features

Topic channels (first preview)

A topic channel is a queue channel that can receive values from multiple sources, based on a matching name or topic.

Topic channels are particularly useful for collecting metadata from various points in a pipeline without writing all of the channel logic that is normally required (e.g., using the mix operator).

See topic for details.

Process `eval` outputs

A process eval output is a new type of process output that captures the standard output of an arbitrary shell command.

This output type is a simpler way to capture command outputs, such as the version of a command-line tool, instead of using a path or env output.

See Eval output (eval) for details.

Resource limits

The resourceLimits process directive allows you to define global limits on the resources requested by individual tasks.

This directive is useful for ensuring that a task’s resource requirements never exceed system-wide limits. It is especially important for preventing tasks from becoming unschedulable when using retry with dynamic resources.

See resourceLimits for details.

Job arrays

The array directive allows a process to submit jobs as job arrays, a feature used by many batch schedulers to submit many similar jobs in an efficient manner.

Tip

On Google Batch, the array directive can be used in conjunction with the cpus, memory, and machineType directives to pack multiple tasks onto the same VM.

See array for details.

Workflow outputs (first preview)

Workflow outputs are a new way to publish outputs at the workflow level:

nextflow.preview.output = true

workflow {
  main:
  ch_foo = foo(data)
  bar(ch_foo)

  publish:
  ch_foo >> 'foo'
}

output {
  directory 'results'
  mode 'copy'
}

This syntax allows you to publish channels in a workflow, instead of publishing file patterns in a process using publishDir.

See Workflow outputs for details.

Enhancements

Colored logs

Nextflow’s console output now features colored logs and improved organization to highlight the most important information about a pipeline while it’s running.

AWS Fargate

Nextflow now supports AWS Fargate for the AWS Batch executor. See AWS Fargate for details.

OCI auto-pull mode for Singularity and Apptainer

The singularity.ociAutoPull (apptainer.ociAutoPull) config option enables OCI auto-pull mode for Singularity (Apptainer). When enabled, this setting allows Singularity (Apptainer) to download and convert Docker images during task execution rather than on the Nextflow head node.

See singularity and apptainer for details.

Improved publish error handling

The publishing logic used by publishDir and workflow outputs will automatically retry failed publish operations with exponential backoff, providing better resilience against transient errors. The retry behavior can be controlled using the nextflow.publish.retryPolicy.* config settings.

Additionally, the failOnError option of the publishDir directive is now true by default. This means that, by default, publish errors (after retries) will cause the pipeline to fail. To restore the previous behavior, set failOnError to false for each publishDir directive.

Note

The nextflow.publish config settings have since been moved to workflow.output. See workflow for details.

GA4GH TES 1.1

The TES executor has been updated to be compliant with version 1.1 of the TES API specification. Notable improvements include support for directory outputs, glob outputs, and the bin directory.