kevin/ma
1
0
Fork 0
Code Issues 1 Packages Activity

add nextflow d30e48d

Browse Source
This commit is contained in:
Kevin
2026-04-29 23:01:54 +02:00
parent d0b12d668d
commit 97cc9058d3
2840 changed files with 730250 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

234
nextflow/docs/modules/developing-modules.md Normal file
View File

@@ -0,0 +1,234 @@
(dev-modules-page)=
# Developing modules
Learn how to create modules and share them through the Nextflow module registry.
(dev-modules-creating)=
## Creating a module
Use the `module create` command to scaffold a new module with the required files:
```console
$ nextflow module create myorg/my-module
```
If you omit the name, the command prompts you for details:
```console
$ nextflow module create
```
The command creates a module directory with the following files:
- `main.nf`: The module script containing your process definition.
- `meta.yml`: The module spec describing metadata, inputs, and outputs.
- `README.md`: Documentation for the module.
See {ref}`module create <cli-module-create>` for the full command reference.
(dev-modules-structure)=
## Module structure
Registry modules follow a standard directory structure:
```
modules/
└── myorg/
└── my-module/
├── .module-info # Integrity checksum
├── README.md # Documentation
├── main.nf # Module script
├── meta.yml # Module spec
├── resources/ # Optional: Module resources
└── templates/ # Optional: Process script templates
```
Local modules that are not intended for publishing do not need to follow this structure, although it is recommended as a best practice.
### main.nf
The `main.nf` file contains the process definition. For example, a simple module wrapping FastQC:
```nextflow
process FASTQC {
tag "$meta.id"
label 'process_medium'
conda 'bioconda::fastqc=0.12.1'
container 'biocontainers/fastqc:0.12.1--hdfd78af_0'
input:
tuple val(meta), path(reads)
output:
tuple val(meta), path("*.html"), emit: html
tuple val(meta), path("*.zip") , emit: zip
script:
"""
fastqc $reads --threads $task.cpus
"""
}
```
Registry modules are subject to the following constraints:
- The module must contain a single script called `main.nf`
- The module must define a single process
- The module must not define any named workflows
- The module may define an entry workflow to override the default behavior of `module run`.
- The module may define any number of functions
Local modules can define any number of processes, workflows, and functions. As a best practice, each process and named workflow should be defined in its own script.
### meta.yml
The `meta.yml` file contains the module's metadata, including its name, version, description, authors, and input/output specifications. The registry uses this file to display module information and generate usage templates.
### README.md
The `README.md` file provides documentation for the module. It should describe what the module does, the tools it wraps, and any configuration requirements.
(module-resources)=
### resources
Modules can include resource files in the `resources/` directory.
When running the module with {ref}`wave-page`, the contents of `resources/` are mounted into the root directory of the task container.
For example, given a module with the following structure:
```
my-module/
├── main.nf
└── resources/
├── data/
| └── file.txt
└── usr/
└── bin/
└── hello.sh
```
The process script can use these files as follows:
```nextflow
process hello {
container 'quay.io/nextflow/bash'
script:
"""
cat /data/file.txt
hello.sh
"""
}
```
Module resources can be used without Wave or containerization, with the following limitations:
- The `nextflow.enable.moduleBinaries` feature flag must be enabled in the pipeline script.
- The pipeline work directory must be in a local or shared file system. Remote object storage is not supported without Wave.
- Only executable scripts in `resources/usr/bin/` are made accessible to the process script.
### templates
Modules can include process script {ref}`templates <process-template>` in the `templates/` directory.
For example, given a module with the following structure:
```
my-module/
|── main.nf
└── templates/
└── hello.sh
```
The module's process can use the script template as follows:
```nextflow
process hello {
input:
val STR
script:
template 'hello.sh'
}
```
(dev-modules-spec)=
## Generating a module spec
Use the `module spec` command to generate or update the `meta.yml` file from the module's `main.nf`:
```console
$ nextflow module spec myorg/my-module
```
Use `-dry-run` to preview the generated spec without writing to disk:
```console
$ nextflow module spec -dry-run myorg/my-module
```
When generating the module spec for the first time, provide required fields directly to avoid `TODO` placeholders in the generated file:
```console
$ nextflow module spec \
-namespace myorg \
-version 1.0.0 \
-description "Quality control of raw sequencing reads" \
-license MIT \
-author "@myname" \
./modules/myorg/my-module
```
When updating an existing module spec, it is incorporated into the new file.
See {ref}`module spec <cli-module-spec>` for the full command reference.
(dev-modules-validate)=
## Validating a module
Use the `module validate` command to check that a module is ready for publishing:
```console
$ nextflow module validate myorg/my-module
```
The command verifies that:
- All required files are present (`main.nf`, `meta.yml`, `README.md`).
- The module spec contains all required fields (name, version, description, and license).
See {ref}`module validate <cli-module-validate>` for the full command reference.
(dev-modules-testing)=
## Testing a module
Before publishing, test your module by running it directly:
```console
$ nextflow module run myorg/my-module --input 'test-data/*.fastq.gz'
```
The command executes the module as a standalone run, allowing you to verify that inputs are correctly declared, the process runs successfully, and the correct outputs are produced.
For more thorough testing, create a small wrapper workflow that exercises the module:
```nextflow
include { MY_MODULE } from './modules/myorg/my-module'
workflow {
input_ch = channel.fromPath('test-data/*.fastq.gz')
results_ch = MY_MODULE(input_ch)
results_ch.view()
}
```

175
nextflow/docs/modules/module-registry.md Normal file
View File

@@ -0,0 +1,175 @@
(module-registry-page)=
# Nextflow module registry
:::{versionadded} 26.04.0
:::
The [Nextflow module registry](https://registry.nextflow.io) is a central repository for sharing Nextflow modules.
It provides module discovery, version tracking, and integrity verification.
The module registry shares its login, access tokens, and web interface with the {ref}`Nextflow plugin registry <plugin-registry-page>`.
(module-registry-claim)=
## Claiming a namespace
A namespace is an organizational prefix (e.g., the namespace `nf-core` owns modules like `nf-core/fastqc` and `nf-core/samtools`).
As a namespace member, you can publish new modules and release new versions under that prefix.
Each namespace can have multiple members who share publishing rights.
To claim a namespace:
1. Open the [Nextflow module registry](https://registry.nextflow.io/) in a browser.
2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address.
:::{note}
If you are logging in for the first time, Seqera sends an authentication link to your email address to complete the login process.
:::
3. Go to **My Resources** and select the **My Namespaces** tab.
4. Select **Claim Namespace**
4. Enter your **Namespace name**, **Organization**, **Project URL**, and a **Description**.
5. Select **Submit Request**.
The namespace shows as **PENDING REVIEW** until an administrator approves the request.
Admin approval is required only once.
Once approved, you can publish modules using the `nextflow module publish` command.
(module-registry-access-token)=
## Creating an access token
An API access token is required to publish modules to the registry.
To create an API access token:
1. Open the [Nextflow module registry](https://registry.nextflow.io/) in a browser.
2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address.
:::{note}
If you are logging in for the first time, Seqera sends an authentication link to your email address to complete the login process.
:::
3. Go to **My Resources** and select the **My access tokens** tab.
4. Under **Create New Access Token**, enter a descriptive name for the **Token name** and select a token duration from the **Expiry** drop down.
5. Select **Generate Token**.
6. Copy the token and store it somewhere safe.
You cannot view it again after this step.
Once you have your token, see [Publishing modules](#publishing-modules) for instructions on how to use it.
## Viewing modules
View the modules you have published in the registry:
1. Open the [Nextflow module registry](https://registry.nextflow.io/) in a browser.
2. Log in and go to **My Resources**.
3. Select the **My Modules** tab.
The page lists all modules published under your namespaces, along with their versions and status.
(publish-modules)=
## Publishing modules
Publishing requires a {ref}`claimed namespace <module-registry-claim>` and an {ref}`access token <module-registry-access-token>`.
Provide your registry token in one of two ways:
**Environment variable:**
```console
$ export NXF_REGISTRY_TOKEN=<token>
```
**Nextflow configuration:**
```groovy
registry {
apiKey = '${MYORG_TOKEN}'
}
```
Use the `module publish` command to upload a module to the registry:
```console
$ nextflow module publish myorg/my-module
```
The module directory must include the following files:
- `main.nf`: The module script.
- `meta.yml`: The module spec with name, version, description, and license.
- `README.md`: Module documentation.
Provide a `namespace/name` reference or a relative path to the module directory.
:::{tip}
Before publishing, verify your module is ready:
- Run {ref}`module spec <cli-module-spec>` to generate or update `meta.yml` from your `main.nf`.
- Run {ref}`module validate <cli-module-validate>` to check that all required files and spec fields are present.
See {ref}`dev-modules-page` for a complete guide on creating modules.
:::
See {ref}`module publish <cli-module-publish>` for the full command reference.
## Registry configuration
By default, Nextflow uses the public registry at `https://registry.nextflow.io`.
Configure alternative or additional registries using the `registry` scope in your Nextflow configuration.
### Use a private registry
Replace the default registry with your organization's private registry:
```groovy
registry {
url = 'https://registry.myorg.com'
apiKey = '${MYORG_TOKEN}'
}
```
Nextflow uses this registry for all module operations (search, install, and publish).
### Use multiple registries
Specify multiple registries as a list.
Nextflow queries them in order when searching for or installing a module:
```groovy
registry {
url = [
'https://registry.myorg.com',
'https://private.registry.nextflow.io'
]
apiKey = '${MYORG_TOKEN}'
}
```
In this example, Nextflow searches the private registry first and falls back to the public registry.
:::{note}
The `apiKey` authenticates with the primary (first) registry.
:::
:::{tip}
Override the target registry with the `-registry` flag:
```console
$ export NXF_REGISTRY_TOKEN=<token>
$ nextflow module publish myorg/my-module -registry 'https://private.registry.myorg.com'
```
:::

63
nextflow/docs/modules/modules.md Normal file
View File

@@ -0,0 +1,63 @@
(modules-page)=
# Overview
Nextflow scripts can include **definitions** (workflows, processes, and functions) from other scripts.
When a script is included in this way, it is referred to as a **module**.
Modules can be re-used within a pipeline and can be shared across projects.
By packaging definitions as modules, you avoid duplicating code and benefit from community improvements.
There are two ways to use modules in Nextflow:
- [Local modules](#local-modules)
- [Registry modules](#registry-modules)
See {ref}`using-modules-page` to learn how to install and use modules in a project.
See {ref}`dev-modules-page` to learn how to create and publish your own modules.
See {ref}`module-registry-page` to learn how to use the Nextflow module registry.
## Local modules
Local modules are stored and maintained directly in your project. You include definitions from local modules using the `include` keyword with a relative path:
```nextflow
include { CAT } from './modules/cat'
workflow {
data = channel.fromPath('data/*.txt')
CAT(data)
}
```
The above snippet imports a process named `CAT` from the *included module* into the *including script*. The include source `./modules/cat` refers to the script `./modules/cat.nf` relative to the including script path.
Local modules are well suited for project-specific components that are not intended for sharing.
## Registry modules
:::{versionadded} 26.04.0
:::
Registry modules are sourced from the [Nextflow registry](https://registry.nextflow.io) or a compatible module registry.
Like local modules, registry modules are stored in your project, but they are installed and managed using the `nextflow module` command.
Key features of registry modules:
- **Discoverability**: Search for modules by keyword or name and browse available versions.
- **Version management**: Pin specific versions or use the latest release, with automatic integrity checking via checksums.
- **Direct execution**: Run modules as standalone workflows for ad-hoc tasks or testing without writing a wrapper script.
- **Standard structure**: Each module includes a script (`main.nf`), spec (`meta.yml`), and documentation (`README.md`), providing a consistent structure for tooling and automation.
:::{note}
Modules from the [nf-core](https://nf-co.re/) community are automatically mirrored to the Nextflow module registry under the `nf-core` namespace.
:::
Install registry modules into your project and include them by name:
```console
$ nextflow module install nf-core/fastqc
```
```nextflow
include { FASTQC } from 'nf-core/fastqc'
```

163
nextflow/docs/modules/using-modules.md Normal file
View File

@@ -0,0 +1,163 @@
(using-modules-page)=
# Using modules
:::{versionadded} 26.04.0
:::
The Nextflow module system allows you to discover, install, and manage reusable modules from centralized registries.
This page describes how to use modules in your pipelines.
## Discovering modules
Search for available modules using the `module search` command:
```console
$ nextflow module search alignment
$ nextflow module search "quality control" -limit 10
```
Results include module names and descriptions.
Use `-output json` for machine-readable output.
See {ref}`module search <cli-module-search>` for the full command reference.
## Installing modules
Use the `module install` command to download modules from a registry into your project:
```console
$ nextflow module install nf-core/fastqc
$ nextflow module install nf-core/fastqc -version 0.0.0-0c7146d
```
:::{note}
Modules mirrored from nf-core do not follow standard semantic versioning.
Instead, they use the format `0.0.0-<hash>`, where the suffix is a short portion of the nf-core module's commit hash.
:::
Nextflow stores installed modules in the `modules/` directory and creates a `.module-info` file alongside the module to record installation metadata such as the module checksum and registry URL.
:::{tip}
Commit the `modules/` directory to your Git repository to ensure reproducibility.
:::
See {ref}`module install <cli-module-install>` for the full command reference.
## Listing installed modules
View all modules installed in your project with the `module list` command:
```console
$ nextflow module list
```
The output shows each module's name, installed version, and whether it has been modified locally.
Use `-output json` for machine-readable output.
See {ref}`module list <cli-module-list>` for the full command reference.
## Viewing module information
Use the `module view` command to view metadata and a usage template for a module:
```console
$ nextflow module view nf-core/fastqc
$ nextflow module view nf-core/fastqc -version 0.0.0-0c7146d
```
The output includes the module's version, URL, description, authors, maintainers, keywords, tools, input/output channels, and a generated usage template.
Use `-output json` for machine-readable output.
See {ref}`module view <cli-module-view>` for the full command reference.
## Including modules
Modules installed from a registry can be included by name:
```nextflow
include { FASTQC } from 'nf-core/fastqc'
workflow {
reads = channel.fromPath('data/*.fastq')
reads = reads.map { fastq -> tuple([id: fastq.baseName], fastq) }
FASTQC(reads)
}
```
Local modules must be included by relative path:
```nextflow
include { FASTQC } from './modules/local/fastqc'
```
See {ref}`syntax-include` for a full description of the include syntax.
## Running modules directly
For ad-hoc tasks or testing, run a module directly without creating a wrapper workflow:
```console
$ nextflow module run nf-core/fastqc --meta.id test_sample --reads sample1_R1.fastq.gz
```
:::{tip}
Run `nextflow module view` to see the available inputs for a module.
:::
The command automatically downloads the module if it is not already installed.
It accepts all standard Nextflow run options (`-profile`, `-resume`, etc.):
```console
$ nextflow module run nf-core/fastqc \
--meta.id test_sample \
--reads sample1_R1.fastq.gz \
-with-docker
```
Run a local module by specifying a path starting with `./` or `../`:
```console
$ nextflow module run ./modules/local/fastqc/main.nf \
--meta.id test_sample \
--reads sample1_R1.fastq.gz \
-with-docker
```
:::{note}
The local module should define a single process, or else the command will fail.
:::
See {ref}`module run <cli-module-run>` for the full command reference.
## Updating modules
To update a module to a newer version, reinstall it with the desired version:
```console
$ nextflow module install nf-core/fastqc -version 0.0.0-c9h0bv4
```
Nextflow automatically verifies module integrity using a checksum stored in the `.module-info` file.
If the module has local modifications, it will not be updated.
Use the `-force` flag to overwrite local changes:
```console
$ nextflow module install nf-core/fastqc -version 0.0.0-c9h0bv4 -force
```
## Removing modules
Use the `module remove` command to uninstall a module from your project:
```console
$ nextflow module remove nf-core/fastqc
```
By default, Nextflow removes both the module files and the `.module-info` file.
Use flags to control this behavior:
- `-keep-files`: Remove the `.module-info` file but keep the module files in the `modules/` directory.
- `-force`: Remove the module directory even if it has no `.module-info` file or has local modifications.
See {ref}`module remove <cli-module-remove>` for the full command reference.