add nextflow d30e48d
This commit is contained in:
136
nextflow/docs/migrations/24-04.md
Normal file
136
nextflow/docs/migrations/24-04.md
Normal file
@@ -0,0 +1,136 @@
|
||||
(migrating-24-04-page)=
|
||||
|
||||
# Migrating to 24.04
|
||||
|
||||
[Nextflow 24.04](https://github.com/nextflow-io/nextflow/releases/tag/v24.04.0) was released on May 20, 2024.
|
||||
|
||||
## New features
|
||||
|
||||
<h3>Topic channels (first preview)</h3>
|
||||
|
||||
A topic channel is a channel that can receive values from multiple sources, based on a matching name or *topic*.
|
||||
|
||||
Topic channels are particularly useful for collecting metadata from various points in a pipeline without writing all of the channel logic that is normally required (e.g., using the `mix` operator).
|
||||
|
||||
See {ref}`channel-topic` for details.
|
||||
|
||||
<h3>Process <code>eval</code> outputs</h3>
|
||||
|
||||
A process `eval` output is a new type of process output that captures the standard output of an arbitrary shell command.
|
||||
|
||||
This output type is a simpler way to capture command outputs, such as the version of a command-line tool, instead of using a `path` or `env` output.
|
||||
|
||||
See {ref}`process-out-eval` for details.
|
||||
|
||||
<h3>Resource limits</h3>
|
||||
|
||||
The `resourceLimits` process directive allows you to define global limits on the resources requested by individual tasks.
|
||||
|
||||
This directive is useful for ensuring that a task's resource requirements never exceed system-wide limits. It is especially important for preventing tasks from becoming unschedulable when using {ref}`retry with dynamic resources <dynamic-task-resources>`.
|
||||
|
||||
See {ref}`process-resourcelimits` for details.
|
||||
|
||||
<h3>Job arrays</h3>
|
||||
|
||||
The `array` directive allows a process to submit jobs as *job arrays*, a feature used by many batch schedulers to submit many similar jobs in an efficient manner.
|
||||
|
||||
:::{tip}
|
||||
On Google Batch, the `array` directive can be used in conjunction with the `cpus`, `memory`, and `machineType` directives to pack multiple tasks onto the same VM.
|
||||
:::
|
||||
|
||||
See {ref}`process-array` for details.
|
||||
|
||||
(workflow-outputs-first-preview)=
|
||||
|
||||
<h3>Workflow outputs (first preview)</h3>
|
||||
|
||||
Workflow outputs are a new way to publish outputs at the workflow level:
|
||||
|
||||
```nextflow
|
||||
nextflow.preview.output = true
|
||||
|
||||
workflow {
|
||||
main:
|
||||
ch_fastqc = fastqc(data)
|
||||
multiqc(ch_fastqc.collect())
|
||||
|
||||
publish:
|
||||
ch_fastqc >> 'qc'
|
||||
multiqc.out >> 'summary'
|
||||
}
|
||||
|
||||
output {
|
||||
directory 'results'
|
||||
mode 'copy'
|
||||
}
|
||||
```
|
||||
|
||||
This syntax allows you to publish channels in a workflow, instead of publishing file patterns in a process using `publishDir`.
|
||||
|
||||
See {ref}`workflow-output-def` for details.
|
||||
|
||||
## Enhancements
|
||||
|
||||
<h3>Colored logs</h3>
|
||||
|
||||
Nextflow's console output now features colored logs and improved organization to highlight the most important information about a pipeline while it's running.
|
||||
|
||||
<h3>AWS Fargate</h3>
|
||||
|
||||
Nextflow now supports AWS Fargate for the AWS Batch executor. See {ref}`aws-fargate` for details.
|
||||
|
||||
<h3>OCI auto-pull mode for Singularity and Apptainer</h3>
|
||||
|
||||
The `singularity.ociAutoPull` (`apptainer.ociAutoPull`) config option enables OCI auto-pull mode for Singularity (Apptainer). When enabled, this setting allows Singularity (Apptainer) to download and convert Docker images during task execution rather than on the Nextflow head node.
|
||||
|
||||
See {ref}`config-singularity` and {ref}`config-apptainer` for details.
|
||||
|
||||
<h3>Improved publish error handling</h3>
|
||||
|
||||
The publishing logic used by `publishDir` and workflow outputs will automatically retry failed publish operations with exponential backoff, providing better resilience against transient errors. The retry behavior can be controlled using the `nextflow.publish.retryPolicy.*` config settings.
|
||||
|
||||
Additionally, the `failOnError` option of the `publishDir` directive is now `true` by default. This means that, by default, publish errors (after retries) will cause the pipeline to fail. To restore the previous behavior, set `failOnError` to `false` for each `publishDir` directive.
|
||||
|
||||
:::{note}
|
||||
The `nextflow.publish` config settings have since been moved to `workflow.output`. See {ref}`config-workflow` for details.
|
||||
:::
|
||||
|
||||
<h3>GA4GH TES 1.1</h3>
|
||||
|
||||
The TES executor has been updated to be compliant with version 1.1 of the [TES API specification](https://ga4gh.github.io/task-execution-schemas/docs/). Notable improvements include support for directory outputs, glob outputs, and the `bin` directory.
|
||||
|
||||
:::{note}
|
||||
The `nf-ga4gh` plugin has since been moved into its own repository, [nextflow-io/nf-ga4gh](https://github.com/nextflow-io/nf-ga4gh). Refer to this repository for documentation and the latest updates on GA4GH integrations for Nextflow.
|
||||
:::
|
||||
|
||||
## Breaking changes
|
||||
|
||||
- Support for {ref}`Glacier auto-retrieval <config-aws>` was removed. Consider using the AWS CLI to restore Glacier objects before or at the beginning of your pipeline.
|
||||
|
||||
- The `seqera` and `defaults` channels were removed from the list of default conda channels used by Wave. They can be restored with the following setting:
|
||||
|
||||
```groovy
|
||||
conda.channels = ['seqera', 'conda-forge', 'bioconda', 'defaults']
|
||||
|
||||
## Miscellaneous
|
||||
|
||||
- New config option: `azure.batch.pools.<name>.lowPriority`
|
||||
- New config option: `azure.batch.pools.<name>.startTask.script`
|
||||
- New config option: `azure.batch.pools.<name>.startTask.privileged`
|
||||
- New config option: `executor.account`
|
||||
- New config option: `fusion.cacheSize`
|
||||
- New config option: `google.batch.maxSpotAttempts`
|
||||
- New config option: `k8s.cpuLimits`
|
||||
- New config option: `k8s.fuseDevicePlugin`
|
||||
- New environment variable: `NXF_CLOUDINFO_ENABLED`
|
||||
- New environment variable: `NXF_CACHE_DIR`
|
||||
- New plugin extension point: `TaskTipProvider`
|
||||
- New `pod` directive option: `schedulerName`
|
||||
- New `pod` directive option: `ttlSecondsAfterFinished`
|
||||
- New workflow metadata: `workflow.fusion.enabled`
|
||||
- New workflow metadata: `workflow.fusion.version`
|
||||
- New workflow metadata: `workflow.preview`
|
||||
- New workflow metadata: `workflow.wave.enabled`
|
||||
- Support Google Batch {ref}`instance templates <google-batch-process>`
|
||||
- Support the use of {ref}`secrets in the pipeline script <secrets-pipeline-script>`
|
||||
- Upgrade to Groovy 4
|
||||
Reference in New Issue
Block a user