ma/nextflow/docs/migrations/24-04.md

(migrating-24-04-page)=

# Migrating to 24.04

[Nextflow 24.04](https://github.com/nextflow-io/nextflow/releases/tag/v24.04.0) was released on May 20, 2024.

## New features

<h3>Topic channels (first preview)</h3>

A topic channel is a channel that can receive values from multiple sources, based on a matching name or *topic*.

Topic channels are particularly useful for collecting metadata from various points in a pipeline without writing all of the channel logic that is normally required (e.g., using the `mix` operator).

See {ref}`channel-topic` for details.

<h3>Process <code>eval</code> outputs</h3>

A process `eval` output is a new type of process output that captures the standard output of an arbitrary shell command.

This output type is a simpler way to capture command outputs, such as the version of a command-line tool, instead of using a `path` or `env` output.

See {ref}`process-out-eval` for details.

<h3>Resource limits</h3>

The `resourceLimits` process directive allows you to define global limits on the resources requested by individual tasks.

This directive is useful for ensuring that a task's resource requirements never exceed system-wide limits. It is especially important for preventing tasks from becoming unschedulable when using {ref}`retry with dynamic resources <dynamic-task-resources>`.

See {ref}`process-resourcelimits` for details.

<h3>Job arrays</h3>

The `array` directive allows a process to submit jobs as *job arrays*, a feature used by many batch schedulers to submit many similar jobs in an efficient manner.

:::{tip}
On Google Batch, the `array` directive can be used in conjunction with the `cpus`, `memory`, and `machineType` directives to pack multiple tasks onto the same VM.
:::

See {ref}`process-array` for details.

(workflow-outputs-first-preview)=

<h3>Workflow outputs (first preview)</h3>

Workflow outputs are a new way to publish outputs at the workflow level:

```nextflow
nextflow.preview.output = true

workflow {
  main:
  ch_fastqc = fastqc(data)
  multiqc(ch_fastqc.collect())

  publish:
  ch_fastqc >> 'qc'
  multiqc.out >> 'summary'
}

output {
  directory 'results'
  mode 'copy'
}
```

This syntax allows you to publish channels in a workflow, instead of publishing file patterns in a process using `publishDir`.

See {ref}`workflow-output-def` for details.

## Enhancements

<h3>Colored logs</h3>

Nextflow's console output now features colored logs and improved organization to highlight the most important information about a pipeline while it's running.

<h3>AWS Fargate</h3>

Nextflow now supports AWS Fargate for the AWS Batch executor. See {ref}`aws-fargate` for details.

<h3>OCI auto-pull mode for Singularity and Apptainer</h3>

The `singularity.ociAutoPull` (`apptainer.ociAutoPull`) config option enables OCI auto-pull mode for Singularity (Apptainer). When enabled, this setting allows Singularity (Apptainer) to download and convert Docker images during task execution rather than on the Nextflow head node.

See {ref}`config-singularity` and {ref}`config-apptainer` for details.

<h3>Improved publish error handling</h3>

The publishing logic used by `publishDir` and workflow outputs will automatically retry failed publish operations with exponential backoff, providing better resilience against transient errors. The retry behavior can be controlled using the `nextflow.publish.retryPolicy.*` config settings.

Additionally, the `failOnError` option of the `publishDir` directive is now `true` by default. This means that, by default, publish errors (after retries) will cause the pipeline to fail. To restore the previous behavior, set `failOnError` to `false` for each `publishDir` directive.

:::{note}
The `nextflow.publish` config settings have since been moved to `workflow.output`. See {ref}`config-workflow` for details.
:::

<h3>GA4GH TES 1.1</h3>

The TES executor has been updated to be compliant with version 1.1 of the [TES API specification](https://ga4gh.github.io/task-execution-schemas/docs/). Notable improvements include support for directory outputs, glob outputs, and the `bin` directory.

:::{note}
The `nf-ga4gh` plugin has since been moved into its own repository, [nextflow-io/nf-ga4gh](https://github.com/nextflow-io/nf-ga4gh). Refer to this repository for documentation and the latest updates on GA4GH integrations for Nextflow.
:::

## Breaking changes

- Support for {ref}`Glacier auto-retrieval <config-aws>` was removed. Consider using the AWS CLI to restore Glacier objects before or at the beginning of your pipeline.

- The `seqera` and `defaults` channels were removed from the list of default conda channels used by Wave. They can be restored with the following setting:

  ```groovy
  conda.channels = ['seqera', 'conda-forge', 'bioconda', 'defaults']

## Miscellaneous

- New config option: `azure.batch.pools.<name>.lowPriority`
- New config option: `azure.batch.pools.<name>.startTask.script`
- New config option: `azure.batch.pools.<name>.startTask.privileged`
- New config option: `executor.account`
- New config option: `fusion.cacheSize`
- New config option: `google.batch.maxSpotAttempts`
- New config option: `k8s.cpuLimits`
- New config option: `k8s.fuseDevicePlugin`
- New environment variable: `NXF_CLOUDINFO_ENABLED`
- New environment variable: `NXF_CACHE_DIR`
- New plugin extension point: `TaskTipProvider`
- New `pod` directive option: `schedulerName` 
- New `pod` directive option: `ttlSecondsAfterFinished` 
- New workflow metadata: `workflow.fusion.enabled`
- New workflow metadata: `workflow.fusion.version`
- New workflow metadata: `workflow.preview`
- New workflow metadata: `workflow.wave.enabled`
- Support Google Batch {ref}`instance templates <google-batch-process>`
- Support the use of {ref}`secrets in the pipeline script <secrets-pipeline-script>`
- Upgrade to Groovy 4