kevin/ma
1
0
Fork 0
Code Issues 1 Packages Activity

add nextflow d30e48d

Browse Source
This commit is contained in:
Kevin
2026-04-29 23:01:54 +02:00
parent d0b12d668d
commit 97cc9058d3
2840 changed files with 730250 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

136
nextflow/docs/migrations/24-04.md Normal file
View File

@@ -0,0 +1,136 @@
(migrating-24-04-page)=
# Migrating to 24.04
[Nextflow 24.04](https://github.com/nextflow-io/nextflow/releases/tag/v24.04.0) was released on May 20, 2024.
## New features
<h3>Topic channels (first preview)</h3>
A topic channel is a channel that can receive values from multiple sources, based on a matching name or *topic*.
Topic channels are particularly useful for collecting metadata from various points in a pipeline without writing all of the channel logic that is normally required (e.g., using the `mix` operator).
See {ref}`channel-topic` for details.
<h3>Process <code>eval</code> outputs</h3>
A process `eval` output is a new type of process output that captures the standard output of an arbitrary shell command.
This output type is a simpler way to capture command outputs, such as the version of a command-line tool, instead of using a `path` or `env` output.
See {ref}`process-out-eval` for details.
<h3>Resource limits</h3>
The `resourceLimits` process directive allows you to define global limits on the resources requested by individual tasks.
This directive is useful for ensuring that a task's resource requirements never exceed system-wide limits. It is especially important for preventing tasks from becoming unschedulable when using {ref}`retry with dynamic resources <dynamic-task-resources>`.
See {ref}`process-resourcelimits` for details.
<h3>Job arrays</h3>
The `array` directive allows a process to submit jobs as *job arrays*, a feature used by many batch schedulers to submit many similar jobs in an efficient manner.
:::{tip}
On Google Batch, the `array` directive can be used in conjunction with the `cpus`, `memory`, and `machineType` directives to pack multiple tasks onto the same VM.
:::
See {ref}`process-array` for details.
(workflow-outputs-first-preview)=
<h3>Workflow outputs (first preview)</h3>
Workflow outputs are a new way to publish outputs at the workflow level:
```nextflow
nextflow.preview.output = true
workflow {
main:
ch_fastqc = fastqc(data)
multiqc(ch_fastqc.collect())
publish:
ch_fastqc >> 'qc'
multiqc.out >> 'summary'
}
output {
directory 'results'
mode 'copy'
}
```
This syntax allows you to publish channels in a workflow, instead of publishing file patterns in a process using `publishDir`.
See {ref}`workflow-output-def` for details.
## Enhancements
<h3>Colored logs</h3>
Nextflow's console output now features colored logs and improved organization to highlight the most important information about a pipeline while it's running.
<h3>AWS Fargate</h3>
Nextflow now supports AWS Fargate for the AWS Batch executor. See {ref}`aws-fargate` for details.
<h3>OCI auto-pull mode for Singularity and Apptainer</h3>
The `singularity.ociAutoPull` (`apptainer.ociAutoPull`) config option enables OCI auto-pull mode for Singularity (Apptainer). When enabled, this setting allows Singularity (Apptainer) to download and convert Docker images during task execution rather than on the Nextflow head node.
See {ref}`config-singularity` and {ref}`config-apptainer` for details.
<h3>Improved publish error handling</h3>
The publishing logic used by `publishDir` and workflow outputs will automatically retry failed publish operations with exponential backoff, providing better resilience against transient errors. The retry behavior can be controlled using the `nextflow.publish.retryPolicy.*` config settings.
Additionally, the `failOnError` option of the `publishDir` directive is now `true` by default. This means that, by default, publish errors (after retries) will cause the pipeline to fail. To restore the previous behavior, set `failOnError` to `false` for each `publishDir` directive.
:::{note}
The `nextflow.publish` config settings have since been moved to `workflow.output`. See {ref}`config-workflow` for details.
:::
<h3>GA4GH TES 1.1</h3>
The TES executor has been updated to be compliant with version 1.1 of the [TES API specification](https://ga4gh.github.io/task-execution-schemas/docs/). Notable improvements include support for directory outputs, glob outputs, and the `bin` directory.
:::{note}
The `nf-ga4gh` plugin has since been moved into its own repository, [nextflow-io/nf-ga4gh](https://github.com/nextflow-io/nf-ga4gh). Refer to this repository for documentation and the latest updates on GA4GH integrations for Nextflow.
:::
## Breaking changes
- Support for {ref}`Glacier auto-retrieval <config-aws>` was removed. Consider using the AWS CLI to restore Glacier objects before or at the beginning of your pipeline.
- The `seqera` and `defaults` channels were removed from the list of default conda channels used by Wave. They can be restored with the following setting:
```groovy
conda.channels = ['seqera', 'conda-forge', 'bioconda', 'defaults']
## Miscellaneous
- New config option: `azure.batch.pools.<name>.lowPriority`
- New config option: `azure.batch.pools.<name>.startTask.script`
- New config option: `azure.batch.pools.<name>.startTask.privileged`
- New config option: `executor.account`
- New config option: `fusion.cacheSize`
- New config option: `google.batch.maxSpotAttempts`
- New config option: `k8s.cpuLimits`
- New config option: `k8s.fuseDevicePlugin`
- New environment variable: `NXF_CLOUDINFO_ENABLED`
- New environment variable: `NXF_CACHE_DIR`
- New plugin extension point: `TaskTipProvider`
- New `pod` directive option: `schedulerName`
- New `pod` directive option: `ttlSecondsAfterFinished`
- New workflow metadata: `workflow.fusion.enabled`
- New workflow metadata: `workflow.fusion.version`
- New workflow metadata: `workflow.preview`
- New workflow metadata: `workflow.wave.enabled`
- Support Google Batch {ref}`instance templates <google-batch-process>`
- Support the use of {ref}`secrets in the pipeline script <secrets-pipeline-script>`
- Upgrade to Groovy 4

66
nextflow/docs/migrations/24-10.md Normal file
View File

@@ -0,0 +1,66 @@
# Migrating to 24.10
[Nextflow 24.10](https://github.com/nextflow-io/nextflow/releases/tag/v24.10.0) was released on October 27, 2024.
## New features
<h3>Language server</h3>
The [Nextflow language server](https://github.com/nextflow-io/language-server) implements the Language Server Protocol (LSP) for Nextflow scripts and config files. It can be integrated with any editor that functions as an LSP client, and is used by the official [Nextflow VS Code extension](https://marketplace.visualstudio.com/items?itemName=nextflow.nextflow). See {ref}`vscode-page` for details.
The language server parses scripts and config files according to the {ref}`Nextflow language specification <syntax-page>`, which is more strict than the Nextflow CLI. See {ref}`strict-syntax-page` for details.
(workflow-outputs-second-preview)=
<h3>Workflow outputs (second preview)</h3>
The second preview of workflow outputs introduces the following breaking changes from the {ref}`previous version <workflow-outputs-first-preview>`:
- The `publish:` section within processes has been removed. Channels should now be published only from workflows, ideally the entry workflow.
- The `directory` output directive has been replaced with the `outputDir` config option and `-output-dir` command line option, which defaults to `results`. Other directives, such as `mode`, have been replaced with config options under `workflow.output.*`.
In other words, only target blocks can be specified in the output block, but target blocks can still specify directives such as `mode`.
- Target names cannot begin or end with a slash (`/`).
See {ref}`workflow-output-def` to learn more about the workflow output definition.
## Enhancements
<h3>Azure Managed Identities</h3>
Nextflow now supports managed identities for the Azure Batch executor. See {ref}`azure-managed-identities` for details.
<h3>Task previous execution trace</h3>
The `task` variable in the process definition has two new properties, `task.previousTrace` and `task.previousException`, which allows a task to access the runtime metadata of the previous attempt. See {ref}`task-previous-execution-trace` for details.
## Breaking changes
- The `all` release of Nextflow has been replaced with the `dist` release, which includes fewer core plugins by default and supports third-party plugins. See {ref}`install-standalone` for details.
- The `nf-ga4gh` plugin has been moved into its own repository, [nextflow-io/nf-ga4gh](https://github.com/nextflow-io/nf-ga4gh). To continue using the TES executor, you must explicitly declare the `nf-ga4gh` plugin in your Nextflow configuration. Refer to the plugin repository for documentation and updates on GA4GH integrations.
- The {ref}`hyperqueue-executor` executor now requires HyperQueue 0.17.0 or later.
- The use of [Grape](https://docs.groovy-lang.org/latest/html/documentation/grape.html) dependencies with the `@Grab` annotation is no longer supported. Code that requires third-party libraries should be migrated to a plugin.
- Automatic spot retries for AWS Batch and Google Batch have been disabled by default. See {ref}`spot-retries-page` for details.
## Deprecations
- The use of `addParams` and `params` clauses in include declarations is deprecated. See {ref}`strict-syntax-page` for details.
## Miscellaneous
- New config option: `aws.client.requesterPays`
- New config option: `google.batch.autoRetryExitCodes`
- New config option: `google.batch.bootDiskImage`
- New config option: `manifest.contributors`
- New config option: `wave.mirror`
- New config option: `wave.scan.mode`
- New config option: `wave.scan.allowedLevels`
- New config option: `workflow.failOnIgnore`
- Rename `nextflow.publish.retryPolicy` config scope to `workflow.output.retryPolicy`

110
nextflow/docs/migrations/25-04.md Normal file
View File

@@ -0,0 +1,110 @@
(migrating-25-04-page)=
# Migrating to 25.04
[Nextflow 25.04](https://github.com/nextflow-io/nextflow/releases/tag/v25.04.0) was released on May 8, 2025.
## New features
<h3>Strict syntax</h3>
The *strict syntax* is a strict parser for Nextflow DSL2 that implements the {ref}`Nextflow language specification <syntax-page>`. Originally introduced by the [Nextflow language server](https://github.com/nextflow-io/language-server) alongside Nextflow 24.10, the strict syntax is now available in the Nextflow CLI.
The strict syntax is disabled by default. Enable it by setting the environment variable `NXF_SYNTAX_PARSER=v2`. See {ref}`strict-syntax-page` for details.
<h3>Linting and formatting</h3>
The `nextflow lint` command checks Nextflow scripts and config files for errors using the {ref}`strict syntax <strict-syntax-page>`. It can also format Nextflow files using the same formatter as the Nextflow language server. See {ref}`cli-lint` for details.
(workflow-outputs-third-preview)=
<h3>Workflow outputs (third preview)</h3>
The third preview of workflow outputs introduces the following breaking changes from the {ref}`previous version <workflow-outputs-second-preview>`:
- The `publish:` section can only be specified in the entry workflow.
- Workflow outputs in the `publish:` section are assigned instead of using the `>>` operator. Output names must be valid identifiers.
- By default, output files are published to the base output directory, rather than a subdirectory corresponding to the output name.
- The syntax for dynamic publish paths has changed. Instead of defining a closure that returns a closure with the `path` directive, the outer closure should use the `>>` operator to publish individual files. See {ref}`workflow-publishing-files` for details.
- The `mapper` index directive has been removed. Use a `map` operator in the workflow body instead.
See {ref}`migrating-workflow-outputs` to get started.
<h3>Topic channels (out of preview)</h3>
{ref}`Topic channels <channel-topic>`, introduced in Nextflow 24.04 as a preview feature, have been brought out of preview, which means that they can be used without the `nextflow.preview.topic` feature flag.
<h3>Data lineage</h3>
This release introduces built-in provenance tracking, also known as *data lineage*. When `lineage.enabled` is set to `true` in your configuration, Nextflow will record every workflow run, task execution, output file, and the links between them.
You can explore this lineage from the command line using the {ref}`cli-lineage` command. Additionally, you can refer to files in the lineage store from a Nextflow script using the `lid://` path prefix as well as the {ref}`channel-from-lineage` channel factory.
See {ref}`data-lineage-page` to get started.
<h3>Simplified plugin development</h3>
The `nextflow plugin create` sub-command creates the scaffold for a Nextflow plugin using the official [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). It is simpler and easier to maintain than the previous plugin boilerplate.
See the {ref}`gradle-plugin-page` guide for details.
## Enhancements
<h3>Improved <code>inspect</code> command</h3>
Previously, the `nextflow inspect` command included all processes that were invoked in a preview run. Now, the `inspect` command includes all processes that are *included* by the entry workflow (directly or indirectly), which has several benefits:
- It includes all processes that could potentially be invoked by the workflow, not just the processes that are invoked for a particular run configuration.
- It's faster, as it doesn't need to evaluate the entry workflow -- only the includes.
- It can be run as `nextflow inspect <PIPELINE>`, without specifying any parameters or config profiles.
See {ref}`cli-inspect` to learn more about the `inspect` command.
<h3>Plugin version ranges</h3>
When specifying a plugin, the plugin version can now be prefixed with `~` to pin the major and minor version while allowing the latest patch release. This syntax makes it possible to pin the plugin version while automatically using new patch releases.
See {ref}`plugins-page` for details.
<h3>Improved publish error handling</h3>
Nextflow will terminate the run if the thread pool responsible for publishing files takes too long. Previously, this timeout was reported as a warning. Now, it is reported as an error.
The previous behavior can be restored with the following config setting:
```groovy
workflow.output.ignoreErrors = true
```
## Breaking changes
- Nextflow now requires Java 17 or newer. See {ref}`install-requirements` for instructions to install Java 17.
- The {ref}`hyperqueue-executor` executor now requires HyperQueue 0.20.0 or later.
## Deprecations
- The process `shell` section has been deprecated. See {ref}`process-shell` for details.
- Nextflow will report a warning if it encounters a process `shell` directive that contains newlines. This warning will become an error in a future release.
- The `-with-weblog` CLI option has been deprecated. See [nextflow-io/nf-weblog](https://github.com/nextflow-io/nf-weblog) to learn how to use the `nf-weblog` plugin.
## Miscellaneous
- New config option: `aws.batch.terminateUnschedulableJobs`
- New config option: `azure.batch.jobMaxWallClockTime`
- New config option: `fusion.snapshots`
- New config option: `google.batch.gcsfuseOptions`
- New config option: `google.batch.networkTags`
- New config option: `workflow.output.copyAttributes`
- New environment variable: `NXF_PLUGINS_ALLOWED`
- New plugin extension point: `TraceObserverFactoryV2`
- New standard library function: `env()`
- Support `disk` directive for Azure Batch

298
nextflow/docs/migrations/25-10.md Normal file
View File

@@ -0,0 +1,298 @@
(migrating-25-10-page)=
# Migrating to 25.10
[Nextflow 25.10](https://github.com/nextflow-io/nextflow/releases/tag/v25.10.0) was released on October 22, 2025.
## New features
<h3>Workflow params</h3>
:::{note}
Typed parameters require the {ref}`strict syntax <strict-syntax-page>`. Set the `NXF_SYNTAX_PARSER` environment variable to `v2` to enable:
```bash
export NXF_SYNTAX_PARSER=v2
```
:::
The `params` block is a new way to declare pipeline parameters in a Nextflow script:
```nextflow
params {
// Path to input data.
input: Path
// Whether to save intermediate files.
save_intermeds: Boolean = false
}
workflow {
println "params.input = ${params.input}"
println "params.save_intermeds = ${params.save_intermeds}"
}
```
This syntax allows you to declare all parameters in one place with explicit type annotations, and it allows Nextflow to validate parameters at runtime.
See {ref}`workflow-typed-params` for details.
(workflow-outputs-final)=
<h3>Workflow outputs (out of preview)</h3>
{ref}`Workflow outputs <workflow-output-def>`, introduced in Nextflow 24.04 as a preview feature, have been brought out of preview.
If you were using a preview version of workflow outputs, you must remove the `nextflow.preview.output` feature flag when upgrading to Nextflow 25.10. Setting this feature flag in 25.10 will cause your pipeline to fail.
The final version adds the ability to specify a type annotation for each output declaration:
```nextflow
workflow {
main:
ch_samples = // ...
publish:
samples = ch_samples
}
output {
samples: Channel<Map> {
path '.'
index {
path 'samples.csv'
}
}
}
```
See {ref}`migrating-workflow-outputs` to get started.
(static-typing-first-preview)=
<h3>Static typing (preview)</h3>
:::{note}
Static typing requires the {ref}`strict syntax <strict-syntax-page>`. Set the `NXF_SYNTAX_PARSER` environment variable to `v2` to enable:
```bash
export NXF_SYNTAX_PARSER=v2
```
:::
Static typing is a new way to write Nextflow pipelines using *type annotations*. Type annotations help document and validate pipeline code.
```nextflow
workflow RNASEQ {
take:
reads: Channel<Path>
index: Value<Path>
main:
samples_ch = QUANT( reads, index )
emit:
samples: Channel<Path> = samples_ch
}
def isSraId(id: String) -> Boolean {
return id.startsWith('SRA')
}
// feature flag required for typed processes
nextflow.preview.types = true
process fastqc {
input:
(id, fastq_1, fastq_2): Tuple<String,Path,Path>
output:
logs = tuple(id, file('fastqc_logs'))
script:
"""
mkdir fastqc_logs
fastqc -o fastqc_logs -f fastq -q ${fastq_1} ${fastq_2}
"""
}
```
The following declarations can be annotated with types:
- Pipeline parameters (the `params` block)
- Workflow takes and emits
- Process inputs and outputs
- Function parameters and returns
- Local variables
- Closure parameters
- Workflow outputs (the `output` block)
Type annotations can refer to any of the {ref}`standard types <stdlib-types>`.
Some types use *generic type parameters* to work with different data types in a type-safe way. For example:
- `List<E>` and `Channel<E>` use the generic type `E` to specify the element type in the list or channel
- `Map<K,V>` uses the generic types `K` for the key type and `V` for the value type in the map
The following examples show types with type parameters:
```nextflow
// List<E> where E is String
def sequences: List<String> = ['ATCG', 'GCTA', 'TTAG']
// List<E> where E is Path
def fastqs: List<Path> = [file('sample1.fastq'), file('sample2.fastq')]
// Map<K,V> where K is String and V is Integer
def readCounts: Map<String,Integer> = [sample1: 1000, sample2: 1500]
// Channel<E> where E is Path
def ch_bams: Channel<Path> = channel.fromPath('*.bam')
```
Type annotations can be appended with `?` to denote values can be `null`:
```nextflow
def x_opt: String? = null
```
In the type system, queue channels are represented as `Channel`, while value channels are represented as `Value`. To make the terminology clearer and more concise, queue channels are now called *dataflow channels* (or simply *channels*), and value channels are now called *dataflow values*. See {ref}`dataflow-page` for more information.
Groovy-style type annotations (e.g., `String x`) are still supported for functions and local variables. However, the language server and `nextflow lint` will automatically convert them to Nextflow-style annotations during code formatting.
:::{warning}
Since Nextflow-style type annotations are new in Nextflow 25.10, formatting code with Groovy-style type annotations will make it incompatible with previous versions of Nextflow. If you need your code to remain compatible with versions prior to 25.10, run the formatter with Nextflow 25.04:
```bash
NXF_VER=25.04.8 nextflow lint -format .
```
:::
See {ref}`migrating-static-types` for details.
<h3>Auth and Launch commands</h3>
The Nextflow CLI has two new commands: `nextflow auth` and `nextflow launch`. Together, they allow you to authenticate with Seqera Platform and launch pipelines from the command line. This approach streamlines the previous workflow of using `nextflow run` with the `-with-tower` option.
```bash
# Authenticate with Seqera Cloud
nextflow auth login
# Check authentication status
nextflow auth status
# Configure Seqera Platform settings
nextflow auth config
# Launch a pipeline in Seqera Platform
nextflow launch nextflow-io/hello
```
See {ref}`cli-auth` and {ref}`cli-launch` for details.
## Enhancements
<h3>Nextflow plugin registry</h3>
Nextflow now uses the [Nextflow plugin registry](https://registry.nextflow.io/) to download plugins in a more efficient and scalable manner.
The legacy plugin index can still be used by setting the `NXF_PLUGINS_REGISTRY_URL` environment variable:
```bash
export NXF_PLUGINS_REGISTRY_URL="https://raw.githubusercontent.com/nextflow-io/plugins/main/plugins.json"
```
:::{note}
Plugins should be updated to publish to the Nextflow plugin registry using the {ref}`Nextflow Gradle plugin <gradle-plugin-page>` instead of the legacy plugin index. See {ref}`plugin-registry-page` for details.
:::
<h3>New syntax for workflow handlers</h3>
:::{note}
The new syntax for workflow handlers require the {ref}`strict syntax <strict-syntax-page>`. Set the `NXF_SYNTAX_PARSER` environment variable to `v2` to enable:
```bash
export NXF_SYNTAX_PARSER=v2
```
:::
The workflow `onComplete` and `onError` handlers were previously defined by calling `workflow.onComplete` and `workflow.onError` in the pipeline script. You can now define handlers as `onComplete` and `onError` sections in an entry workflow:
```nextflow
workflow {
main:
// ...
onComplete:
println "workflow complete"
onError:
println "error: ${workflow.errorMessage}"
}
```
This syntax is simpler and easier to use with the {ref}`strict syntax <strict-syntax-page>`. See {ref}`workflow-handlers` for details.
<h3>Simpler syntax for dynamic directives</h3>
The {ref}`strict syntax <strict-syntax-page>` allows dynamic process directives to be specified without a closure:
```nextflow
process hello {
queue (entries > 100 ? 'long' : 'short')
input:
tuple val(entries), path('data.txt')
script:
"""
your_command --here
"""
}
```
Dynamic process settings in configuration files must still be specified with closures.
See {ref}`dynamic-directives` for details.
<h3>Configurable date formatting</h3>
You can now customize the date and time format used in notifications, reports, and console output using the `NXF_DATE_FORMAT` environment variable:
```bash
# Default format
# e.g., 11-Aug-2016 09:40:20
# Use ISO format with timezone
export NXF_DATE_FORMAT="iso"
# e.g., 2016-08-11T09:40:20+02:00
# Use custom format
export NXF_DATE_FORMAT="yyyy-MM-dd HH:mm"
# e.g., 2016-08-11 09:40
```
This feature addresses previous inconsistencies in timestamp representations.
## Breaking changes
- The `google-lifesciences` executor is no longer supported because the Google Cloud Life Sciences API was discontinued in July 2025. Use the `google-batch` executor instead.
- The AWS Java SDK used by Nextflow was upgraded from v1 to v2, which introduced some breaking changes to the `aws.client` config options. See {ref}`the guide <aws-java-sdk-v2-page>` for details.
- The `nextflow.config.schema` package was renamed to `nextflow.config.spec`. Plugin developers that define custom {ref}`configuration scopes <dev-plugins-extension-points-config>` will need to update their imports accordingly.
## Deprecations
- The legacy type detection of CLI parameters is disabled when using the strict syntax (`NXF_SYNTAX_PARSER=v2`). {ref}`Legacy parameters <workflow-params-legacy>` in the strict syntax should not rely on legacy type detection. Alternatively, use the new `params` block to convert CLI parameters based on their type annotations. Legacy type detection can be disabled globally by setting the environment variable `NXF_DISABLE_PARAMS_TYPE_DETECTION=true`.
- The use of workflow handlers in the configuration file has been deprecated. You should define workflow handlers in the pipeline script or a plugin instead. See {ref}`config-workflow-handlers` for details.
- The `nextflow.enable.configProcessNamesValidation` feature flag is deprecated. It was originally introduced to suppress false warnings for process selectors targeting conditional processes. The {ref}`strict syntax <strict-syntax-page>` now validates process selectors without producing false warnings.
## Miscellaneous
- New config option: `docker.registryOverride`
- New config options: `wave.build.compression.*`
- New executor: Fujitsu Technical Computing Suite (`tcs`)
- Support Java 25

402
nextflow/docs/migrations/26-04.md Normal file
View File

@@ -0,0 +1,402 @@
(migrating-26-04-page)=
# Migrating to 26.04
[Nextflow 26.04](https://github.com/nextflow-io/nextflow/releases/tag/v26.04.0) was released on April 29, 2026.
## New features
<h3>Module system</h3>
Nextflow now has native support for publishing, installing, running, and including remote modules using the [Nextflow registry](https://registry.nextflow.io/).
The new `module` command can be used to work with remote modules:
```bash
# Run a module directly
nextflow module run nf-core/fastqc --meta.id 1 --reads sample1.fastq
# Search for modules in registry
nextflow module search bwa
# View info about a module
nextflow module view nf-core/bwa/mem
# Install a module
nextflow module install nf-core/bwa/mem
```
Remote modules are installed into the `modules` directory of your project. They can be included in scripts by name instead of a relative path:
```nextflow
// before
include { BWA_MEM } from '../../../modules/nf-core/bwa/mem'
// after
include { BWA_MEM } from 'nf-core/bwa/mem'
```
:::{note}
All [nf-core modules](https://github.com/nf-core/modules) are automatically synced to the registry from GitHub. They are available as `nf-core/<name>` as shown above.
:::
See the {ref}`Modules overview <modules-page>` and {ref}`cli-module` command reference for details.
<h3>Record types</h3>
Records are a new data structure for modeling composite data. They serve as an alternative to tuples. While tuple elements are accessed by index, record fields are accessed by name. This allows you to model data with meaningful names rather than keeping track of the order of tuple elements.
A record can be created using the `record()` function:
```nextflow
sample = record(
id: '1',
fastq_1: file('1_1.fastq'),
fastq_2: file('1_2.fastq')
)
println sample.id
println sample.foo // error: unrecognized property `foo`
```
Record types are a way to define custom types for validating records:
```nextflow
record Sample {
id: String
fastq_1: Path
fastq_2: Path?
}
```
While records created with the `record()` function can have any set of fields, record types can be used to specify a minimum set of requirements in a particular context, such as a process, function, or workflow:
```nextflow
def hello(sample: Sample) {
println "Hello sample ${sample.id}!"
}
workflow {
sample1 = record(id: '1', fastq_2: file('1_2.fastq'))
hello(sample1) // error: `sample1` is missing `fastq_1` field required by Sample
sample2 = record(id: '1', fastq_1: file('1_1.fastq'))
hello(sample2) // ok
sample3 = record(id: '1', single_end: true, fastq_1: file('1_1.fastq'))
hello(sample3) // also ok
}
```
See {ref}`script-records` for details. See {ref}`migrating-static-types` for more information about migrating from tuples to records.
<h3>Static typing (preview)</h3>
:::{note}
Typed processes and typed workflows require the `nextflow.enable.types` feature flag to be enabled in every script that uses them.
:::
Nextflow 26.04 brings full support for static typing with {ref}`typed processes <process-typed-page>` and {ref}`typed workflows <workflow-typed-page>`.
Typed processes can now declare record inputs and outputs:
```nextflow
nextflow.enable.types = true
process FASTQC {
input:
record(
id: String,
fastq_1: Path,
fastq_2: Path
)
output:
record(
id: id,
fastqc: file('fastqc_logs')
)
script:
// ...
}
```
Typed workflows can declare typed inputs and outputs, and provide first-class support for static typing in dataflow logic:
```nextflow
nextflow.enable.types = true
workflow RNASEQ {
take:
read_pairs_ch: Channel<Sample>
transcriptome: Path
main:
index = INDEX(transcriptome)
fastqc_ch = FASTQC(read_pairs_ch)
quant_ch = QUANT(read_pairs_ch, index)
samples_ch = fastqc_ch.join(quant_ch, by: 'id')
emit:
samples: Channel<AlignedSample> = samples_ch
}
record Sample { /* ... */ }
record AlignedSample { /* ... */ }
```
Several operators have been extended to support static typing and records:
- `combine` can now combine records or tuples
- `groupBy` is a statically-typed alternative to `groupTuple`
- `join` can now join records by matching field
Breaking changes from the {ref}`first preview <static-typing-first-preview>`:
- The `nextflow.preview.types` feature flag has been replaced with `nextflow.enable.types`.
- The `nextflow.enable.types` feature flag must be enabled in order to use type annotations in workflow takes/emits.
- The syntax for process tuple inputs has been updated:
```nextflow
// 25.10
input:
(id, fastq_1, fastq_2): Tuple<String,Path,Path>
// 26.04
input:
tuple(id: String, fastq_1: Path, fastq_2: Path)
```
- The method signature for the `stageAs` directive was changed from `(filePattern, value)` to `(value, filePattern)`.
See {ref}`migrating-static-types` for more information about migrating to static typing. See {ref}`migrating-static-types-operators` for best practices on using operators with static typing.
<h3>Process <code>hints</code> directive</h3>
The `hints` process directive allows you to specify executor-specific hints for individual processes.
For example, the AWS Batch executor uses the `consumableReources` hint for [resource-aware scheduling](https://aws.amazon.com/about-aws/whats-new/2025/02/aws-batch-resource-aware-scheduling/):
```nextflow
process hello {
hints 'awsbatch/consumableResources': ['my-license': 1]
// ...
}
```
Custom executors can use hints to control scheduling behavior in a fine-grained manner over configuration options.
See {ref}`process-hints` for details.
## Enhancements
<h3>Strict syntax parser enabled by default</h3>
The {ref}`strict syntax parser <strict-syntax-page>` is now enabled by default. This change adds the rich error checking from `nextflow lint` to `nextflow run`, and makes it easier to adopt new language features.
The legacy parser can be re-enabled by setting the `NXF_SYNTAX_PARSER` environment variable to `v1`.
<h3>Agent logging mode</h3>
Nextflow prints logs in an agent-friendly format when enabled via `NXF_AGENT_MODE=1`:
```console
$ NXF_AGENT_MODE=1 nextflow run [...]
[PIPELINE] nf-core/rnaseq 3.14.0 | profile=test,docker
[WORKDIR] /path/to/work
[PROCESS b8/623fc3] sayHello (3)
[PROCESS 5f/761e51] sayHello (2)
[WARN] Task runtime metrics are not reported when using macOS
[ERROR] FASTQC (sample_1)
exit: 127
cmd: fastqc --version
stderr: bash: fastqc: command not found
workdir: /path/to/work/ab/123456
[SUCCESS] completed=10 failed=0 cached=5
```
Agent mode provides minimal structured logging for efficient token usage. It is ideal when using agents to run and debug Nextflow runs.
<h3>Multi-revision checkout for pipelines</h3>
Nextflow now checks out pipelines separately by revision when running remote pipelines, allowing you to run multiple revisions of a pipeline at the same time.
Nextflow uses the default branch specified by the Git repository when no revision is specified, rather than defaulting to `master`. Since each revision is checked out separately, using a specific revision doesn't prevent the default branch from being used on subsequent runs.
The `manifest.defaultBranch` config option is no longer needed because of this update.
See {ref}`workflow-typed-params` for details. See {ref}`migrating-static-types` for a migration example.
<h3>Print workflow outputs on run completion</h3>
When the `output` block is defined, Nextflow prints a summary of the workflow outputs at the end of a run.
For example, given the following output block:
```nextflow
workflow {
// ...
}
output {
samples {
path { sample -> "${sample.id}" }
index {
path 'samples.csv'
}
}
multiqc_report {
path 'multiqc_report.html'
}
}
```
Nextflow prints the following summary:
```console
$ nextflow -q run main.nf
Outputs:
/path/to/results
samples: samples.csv
multiqc_report: multiqc_report.html
```
You can specify `-output-format json` to produce JSON output instead:
```console
$ nextflow -q run main.nf -output-format json
{
"samples": "/path/to/results/samples.csv",
"multiqc_report": "/path/to/results/multiqc_report.html"
}
```
The `module run` command prints a summary of the module outputs:
```console
$ nextflow -q module run nf-core/fastqc --meta.id 1 --reads sample1.fastq -output-format json
{
"html": [
{
"id": "SRR6357070_1"
},
"/path/to/results/SRR6357070_1_fastqc.html"
],
"zip": [
{
"id": "SRR6357070_1"
},
"/path/to/results/SRR6357070_1_fastqc.zip"
]
}
```
See the {ref}`cli-run` command reference for details.
<h3>New <code>-project-dir</code> option for <code>lint</code> command</h3>
The `lint` command now has a `-project-dir` option, which can be used to specify the project root when linting files.
For example, given the following project structure:
```
├── lib
│ └── Utils.groovy
├── modules
│ └── nf-core
│ └── fastqc
│ ├── main.nf
│ └── ...
├── workflows
│ └── rnaseq.nf
├── main.nf
└── nextflow.config
```
You can specify `-project-dir` when linting from a location other than the project root:
```bash
cd workflows
nextflow lint -project-dir ../ rnaseq.nf
```
This option allows the linter to resolve remote module includes and Groovy code in the `lib` directory:
```nextflow
// workflows/rnaseq.nf
include { FASTQC } from 'nf-core/fastqc'
workflow RNASEQ {
// ...
Utils.validateInput( /* ... */ )
// ...
}
```
See the {ref}`cli-lint` command reference for details.
<h3>Apple container support</h3>
Nextflow can run containers on macOS using [Apple container](https://github.com/apple/container), a lightweight container runtime for Apple silicon machines:
```groovy
process.container = 'nextflow/examples:latest'
appleContainer.enabled = true
```
See {ref}`container-apple` for details.
<h3>Seqera filesystem for datasets</h3>
Nextflow can download [Seqera Platform datasets](https://docs.seqera.io/platform-cloud/data/datasets) using the `seqera` URI scheme.
Seqera dataset URIs are specified as `seqera://<organization>/<workspace>/datasets/<name>[@version]`. For example:
```nextflow
params.dataset = 'seqera://seqeralabs/showcase/datasets/sarek_samples'
workflow {
println file(params.dataset).text
}
```
A Platform access token with appropriate permissions is required to download non-public datasets. It can be specified using the `TOWER_ACCESS_TOKEN` environment variable or the `tower.accessToken` config option.
## Breaking changes
- The {ref}`strict syntax parser <strict-syntax-page>` is now enabled by default. The legacy parser can be enabled by setting the `NXF_SYNTAX_PARSER` environment variable to `v1`.
## Deprecations
- The `nextflow.enable.strict` feature flag is deprecated. It is not needed when {ref}`strict syntax <strict-syntax-page>` is enabled.
- The `manifest.defaultBranch` config option is deprecated. Nextflow can automatically detect the default branch when using a remote pipeline.
- The Path `listFiles()` method is deprecated. Use `listDirectory()` instead.
- Several operators have been deprecated. See {ref}`operator-page` for details.
## Miscellaneous
- New config option: `aws.batch.forceGlacierTransfer`
- New config option: `executor.onlyJobState`
- New config option: `google.batch.installOpsAgent`
- New config option: `google.batch.logsPath`
- New config option: `wave.build.conda.baseImage`
- New config option: `wave.build.template`
- New standard library function: `listDirectory()`
- Support Java 26

203
nextflow/docs/migrations/dsl1.md Normal file
View File

@@ -0,0 +1,203 @@
(dsl1-page)=
# Migrating from DSL1
In Nextflow version `22.03.0-edge`, DSL2 became the default DSL version. In version `22.12.0-edge`, DSL1 support was removed and the Nextflow documentation was updated to use DSL2 by default. Users who are still using DSL1 should migrate their pipelines to DSL2 to use the latest versions of Nextflow. This page describes the differences between DSL1 and DSL2 and how to migrate to DSL2.
## Processes and workflows
In DSL1, a process definition is also the process invocation. Process inputs and outputs are connected to channels using `from` and `into`. You can see a basic Nextflow script written in DSL1 here:
```nextflow
nextflow.enable.dsl=1
params.str = 'Hello world!'
process split_letters {
output:
file 'chunk_*' into letters
script:
"""
printf '${params.str}' | split -b 6 - chunk_
"""
}
process convert_to_upper {
input:
file x from letters.flatten()
output:
stdout result
script:
"""
cat $x | tr '[a-z]' '[A-Z]'
"""
}
result.view { it.trim() }
```
To migrate this code to DSL2, you need to move all of your channel logic throughout the script into a `workflow` definition. Additionally, you must call each process explicitly, passing any input channels as arguments (instead of `from ...`) and receiving any output channels as return values (instead of `into ...`).
See {ref}`workflow-page` page to learn how to define a workflow.
You can see the DSL1 Nextflow script from above written in DSL2 here:
```nextflow
params.str = 'Hello world!'
process split_letters {
output:
path 'chunk_*'
script:
"""
printf '${params.str}' | split -b 6 - chunk_
"""
}
process convert_to_upper {
input:
path x
output:
stdout
script:
"""
cat $x | tr '[a-z]' '[A-Z]'
"""
}
workflow {
split_letters | flatten | convert_to_upper | view { v -> v.trim() }
}
```
## Channel forking
In DSL1, a channel can be used as an input only once; to use a channel multiple times, the channel must be forked using the `into` operator. In DSL2, channels are automatically forked when connecting two or more consumers. For example:
```nextflow
Channel
.of('Hello','Hola','Ciao')
.set { cheers }
cheers
.map { v -> v.toUpperCase() }
.view()
cheers
.map { v -> v.reverse() }
.view()
```
Similarly, in DSL2, process outputs can be consumed by multiple consumers automatically, which makes workflow scripts much easier to read and write.
## Modules
In DSL1, the entire Nextflow pipeline must be defined in a single file. For example, `main.nf`. This restriction becomes cumbersome as a pipeline grows and hinders the sharing and reuse of pipeline components.
DSL2 introduces the concept of "module scripts" (or "modules" for short), which are Nextflow scripts that can be "included" by other scripts. While modules are not essential to migrating to DSL2, nor are they mandatory in DSL2, modules can help you organize a large pipeline into multiple smaller files and take advantage of modules created by others. See {ref}`modules-page` to learn more about modules.
:::{note}
DSL2 scripts cannot exceed 64 KB in size. Split large DSL1 scripts into modules to avoid this limit.
:::
## Deprecations
<h3>Processes</h3>
- The `set` process input type is no longer supported, use {ref}`tuple <process-input-tuple>` instead.
- The `set` process output type is no longer supported, use {ref}`tuple <process-out-tuple>` instead.
- The `mode flatten` option for process outputs is no longer available. Use the {ref}`operator-flatten` operator on the corresponding output channel instead.
- Unqualified value and file elements in a tuple declaration are no longer allowed. Use an explicit `val` or `path` qualifier. For example:
```nextflow
process sam2bam {
input:
tuple X, 'some-file.sam'
output:
tuple X, 'some-file.bam'
script:
"""
your_command --in $X some-file.sam > some-file.bam
"""
}
```
Use:
```nextflow
process sam2bam {
input:
tuple val(X), path('some-file.sam')
output:
tuple val(X), path('some-file.bam')
script:
"""
your_command --in $X some-file.sam > some-file.bam
"""
}
```
<h3>Channels</h3>
- Channel factories should be accessed through the `channel` namespace instead of the `Channel` type, although they can still be accessed through either method.
- Channel method `bind` has been deprecated in DSL2.
- Channel method `<<` has been deprecated in DSL2.
- Channel factory `create` has been deprecated in DSL2.
<h3>Operators</h3>
- Operator `choice` has been deprecated in DSL2. Use {ref}`operator-branch` instead.
- Operator `close` has been deprecated in DSL2.
- Operator `countBy` has been deprecated in DSL2.
- Operator `into` has been deprecated in DSL2, as it is no longer needed.
- Operator `fork` has been renamed to {ref}`operator-multimap`.
- Operator `groupBy` has been deprecated in DSL2. Use {ref}`operator-grouptuple` instead.
- Operators `print` and `println` have been deprecated in DSL2. Use {ref}`operator-view` instead.
- Operator `route` has been deprecated in DSL2.
- Operator `separate` has been deprecated in DSL2.
- Operator `spread` has been deprecated in DSL2. Use {ref}`operator-combine` instead.
<h3>DSL2 Preview</h3>
An early preview of DSL2 was available in 2020. Note that some of that early DSL2 syntax has since changed.
- The `nextflow.preview.dsl=2` (and `nextflow.enable.dsl=1`) feature flags are no longer needed.
- Anonymous and unwrapped includes are no longer supported. Use an explicit module inclusion instead.
For example:
```nextflow
include './modules/hello'
include bye from './modules/bye'
workflow {
hello()
bye()
}
```
Should be replaced with:
```nextflow
include { hello } from './modules/hello'
include { bye } from './modules/bye'
workflow {
hello()
bye()
}
```

16
nextflow/docs/migrations/index.md Normal file
View File

@@ -0,0 +1,16 @@
(migrations-page)=
# Migration notes
This section covers important information for migrating between Nextflow versions. It describes new features, enhancements, breaking changes, deprecations, and other miscellaneous updates that may require action during a migration.
```{toctree}
:maxdepth: 1
26-04
25-10
25-04
24-10
24-04
dsl1
```