kevin/ma
1
0
Fork 0
Code Issues 1 Packages Activity

add nextflow d30e48d

Browse Source
This commit is contained in:
Kevin
2026-04-29 23:01:54 +02:00
parent d0b12d668d
commit 97cc9058d3
2840 changed files with 730250 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

524
nextflow/docs/plugins/developing-plugins.md Normal file
View File

@@ -0,0 +1,524 @@
(dev-plugins-page)=
# Developing plugins
This page describes how to develop plugins for Nextflow.
(dev-plugins-template)=
## Nextflow plugin template
The [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/) is a scaffold for plugin development. It uses [Gradle](https://gradle.org/), a build automation tool optimized for Java and Groovy projects, as well as the [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle).
You can use the `nextflow plugin create` sub-command to create plugins using the plugin template. See {ref}`gradle-plugin-create` for more information.
(dev-plugins-structure)=
### Structure
The plugin template includes the source directories, build configuration files, and metadata required for development, testing, and publishing. Depending on the developer’s preference, plugins can be written in Java or Groovy.
For example, a plugin created from the plugin template with the name `nf-hello` and organization `nextflow` will have the following structure:
```console
nf-hello
├── .github/workflows/build.yml
├── COPYING
├── Makefile
├── README.md
├── build.gradle
├── gradle
│ └── wrapper
│ ├── gradle-wrapper.jar
│ └── gradle-wrapper.properties
├── gradlew
├── settings.gradle
├── src
│ ├── main
│ │ └── groovy
│ │ └── nextflow
│ │ └── hello
│ │ ├── HelloExtension.groovy
│ │ ├── HelloFactory.groovy
│ │ ├── HelloObserver.groovy
│ │ └── HelloPlugin.groovy
│ └── test
│ └── groovy
│ └── nextflow
│ └── hello
│ └── HelloObserverTest.groovy
└── validation
└── main.nf
└── nextflow.config
```
This structure contains the following key files and folders:
- `.github/workflows/build.yml`: GitHub Action which implements continuous integration for the plugin.
- `build.gradle`: The Gradle build script.
- `COPYING`: The project license, detailing the terms under which the code can be used and distributed.​
- `gradle/wrapper/`: Helper files for the Gradle Wrapper.
- `gradlew`: The Gradle Wrapper script, which allows you to use Gradle without installing it into your environment.
- `Makefile`: Defines common tasks for building, testing, and publishing the plugin with Make.​
- `README.md`: The project README, which provides an overview of the project, including its purpose, features, and instructions for usage and development.​
- `settings.gradle`: The Gradle project configuration, which specifies project-specific settings such as the project name and included modules.
- `src/main/groovy/<ORGANIZATION>/`: The main source directory, which contains the plugin source code and resources.​
- `src/test/groovy/<ORGANIZATION>/`: The test source directory, which contains the plugin unit tests.
- `validation`: A small Nextflow pipeline which serves as an end-to-end test for the plugin.
The plugin template also implements the following example features:
- A custom trace observer that prints a message when the workflow starts and when the workflow completes (see `HelloObserver`).
- A custom function called `sayHello` (see `HelloExtension`).
### Nextflow Gradle plugin
The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) simplifies the development of Nextflow plugins. It provides default configuration required for Nextflow integration, as well as custom Gradle tasks for building, testing, and publishing plugins.
It is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), and can be declared and managed like any other dependency in the `build.gradle` file:
```nextflow
plugins {
id 'io.nextflow.nextflow-plugin' version '1.0.0-beta.6'
}
```
:::{note}
You can develop Nextflow plugins without the Gradle plugin. However, this approach is only suggested if you are an advanced developer and your project is incompatible with the Gradle plugin.
:::
### Make commands
The plugin template includes a Makefile which wraps the most important Gradle tasks provided by the Nextflow Gradle plugin.
These tasks can be executed with [Make](https://www.gnu.org/software/make/). For example:
```bash
make assemble
```
The following `make` commands are available:
`assemble`
: Compiles the Nextflow plugin code and assembles it into a zip archive.
`install`
: Installs the plugin into the local Nextflow plugins directory.
`release`
: Publishes the plugin. See {ref}`gradle-plugin-publish` for more information.
`test`
: Runs plugin unit tests. See {ref}`gradle-plugin-test` for more information.
(dev-plugins-extension-points)=
## Extension points
Nextflow’s plugin system exposes various extension points. This section gives examples of typical extension points and how to use them.
### Commands
Plugins can define custom CLI commands that are executable with the `nextflow plugin` command.
To implement a plugin-specific command, implement the `PluginExecAware` interface in your plugin entry point (the class that extends `BasePlugin`). Alternatively, implement the `PluginAbstractExec` trait, which provides an abstract implementation with some boilerplate code. This trait requires you to implement the `getCommands()` and `exec()` methods. For example:
```groovy
import nextflow.cli.PluginAbstractExec
import nextflow.plugin.BasePlugin
class MyPlugin extends BasePlugin implements PluginAbstractExec {
@Override
List<String> getCommands() {
[ 'hello' ]
}
@Override
int exec(String cmd, List<String> args) {
if( cmd == 'hello' ) {
println "Hello! You gave me these arguments: ${args.join(' ')}"
return 0
}
else {
System.err.println "Invalid command: ${cmd}"
return 1
}
}
}
```
The command can be run using the `nextflow plugin` command:
```bash
nextflow plugin my-plugin:hello --alpha --beta
```
See the {ref}`cli-plugin` for usage information.
(dev-plugins-extension-points-config)=
### Configuration
Plugins can access the resolved Nextflow configuration through the session object using `session.config.navigate()`. Several extension points provide the session object for this reason. This method allows you to query any configuration option safely. If the option isn’t defined, it will return null.
A common practice is to use a custom config scope to define any configuration for your plugin. For example:
```groovy
import nextflow.Session
import nextflow.trace.TraceObserver
class MyObserver implements TraceObserver {
@Override
void onFlowCreate(Session session) {
final message = session.config.navigate('myplugin.createMessage')
println message
}
}
```
This option can then be set in your configuration file:
```groovy
// dot syntax
myplugin.createMessage = "I'm alive!"
// block syntax
myplugin {
createMessage = "I'm alive!"
}
```
:::{versionadded} 25.04.0
:::
:::{versionchanged} 25.10.0
The `nextflow.config.schema` package was renamed to `nextflow.config.spec`.
:::
Plugins can declare their configuration options by implementing the `ConfigScope` interface and declaring each config option as a field with the `@ConfigOption` annotation. For example:
```groovy
import nextflow.config.spec.ConfigOption
import nextflow.config.spec.ConfigScope
import nextflow.config.spec.ScopeName
import nextflow.script.dsl.Description
@ScopeName('myplugin')
@Description('''
The `myplugin` scope allows you to configure the `nf-myplugin` plugin.
''')
class MyPluginConfig implements ConfigScope {
@ConfigOption
@Description('''
Message to print to standard output when a run is initialized.
''')
final String createMessage
// no-arg constructor is required to enable validation of config options
MyPluginConfig() {
}
MyPluginConfig(Map opts) {
this.createMessage = opts.createMessage
}
}
```
This approach is not required to support plugin config options. However, it allows Nextflow to recognize plugin definitions when validating the configuration. See {ref}`config-scopes-page` for more information.
### Executors
Plugins can define custom executors that can be used with the `executor` process directive.
To implement an executor, create a class in your plugin that extends the [`Executor`](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/executor/Executor.groovy) class and implements the `ExtensionPoint` interface. Add the `@ServiceName` annotation to your class with the name of your executor. For example:
```groovy
import nextflow.executor.Executor
import nextflow.util.ServiceName
import org.pf4j.ExtensionPoint
@ServiceName('my-executor')
class MyExecutor extends Executor implements ExtensionPoint {
// ...
}
```
You can then use this executor in your pipeline:
```groovy
process hello {
executor 'my-executor'
// ...
}
```
:::{tip}
See the source code of Nextflow's built-in executors for examples of how to implement various executor components.
:::
### Filesystems
Plugins can define custom filesystems that Nextflow can use to interact with external storage systems using a single interface. For more information about accessing remote files, see {ref}`remote-files`.
To implement a custom filesystem, create a class in your plugin that extends [`FileSystemProvider`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/spi/FileSystemProvider.html). Implement the `getScheme()` method to define the URI scheme for your filesystem. For example:
```groovy
import java.nio.file.spi.FileSystemProvider
class MyFileSystemProvider extends FileSystemProvider {
@Override
String getScheme() {
return 'myfs'
}
// ...
}
```
You can then use this filesystem in your pipeline:
```nextflow
input = file('myfs://<PATH_TO_INPUT_FILE>')
```
See [Developing a Custom File System Provider](https://docs.oracle.com/javase/8/docs/technotes/guides/io/fsp/filesystemprovider.html) for more information and the `nf-amazon` plugin (`S3FileSystemProvider`) for an example of a custom filesystem.
:::{tip}
Custom filesystems are an advanced plugin extension. Before creating a new filesystem, check that your use case cannot already be supported by an existing filesystem such as HTTP or S3.
:::
### Functions
:::{versionadded} 22.10.0
:::
Plugins can define custom functions that can be included in Nextflow pipelines.
To implement a custom function, create a plugin class that extends the `PluginExtensionPoint` class and implement your function with the `Function` annotation. For example:
```groovy
import nextflow.Session
import nextflow.plugin.extension.Function
import nextflow.plugin.extension.PluginExtensionPoint
class MyExtension extends PluginExtensionPoint {
@Override
void init(Session session) {}
@Function
String reverseString(String origin) {
origin.reverse()
}
}
```
You can then add this function to your pipeline:
```nextflow
include { reverseString } from 'plugin/my-plugin'
channel.of( reverseString('hi') )
```
Alternatively, you can use an alias:
```nextflow
include { reverseString as anotherReverseMethod } from 'plugin/my-plugin'
```
### Operators
:::{versionadded} 22.04.0
:::
Plugins can define custom channel factories and operators that can then be included in pipelines.
To implement a custom channel factory or operator, create a class in your plugin that extends the `PluginExtensionPoint` class and implement your function with the `Factory` or `Operator` annotation. For example:
```groovy
import groovyx.gpars.dataflow.DataflowReadChannel
import groovyx.gpars.dataflow.DataflowWriteChannel
import nextflow.Session
import nextflow.plugin.extension.Factory
import nextflow.plugin.extension.Operator
import nextflow.plugin.extension.PluginExtensionPoint
class MyExtension extends PluginExtensionPoint {
@Override
void init(Session session) {}
@Factory
DataflowWriteChannel fromQuery(Map opts, String query) {
// ...
}
@Operator
DataflowWriteChannel sqlInsert(DataflowReadChannel source, Map opts) {
// ...
}
}
```
You can then use the custom channel factories or operators in your pipeline:
```nextflow
include { sqlInsert; fromQuery as fromTable } from 'plugin/nf-sqldb'
def sql = 'select * from FOO'
channel
.fromTable(sql, db: 'test', emitColumns: true)
.sqlInsert(into: 'BAR', columns: 'id', db: 'test')
```
:::{note}
The above snippet is based on the [nf-sqldb](https://github.com/nextflow-io/nf-sqldb) plugin. The `fromQuery` factory is included under the alias `fromTable`.
:::
:::{tip}
Before creating a custom operator, consider whether the operator can be defined as a [function](#functions) that can be composed with existing operators such as `map` or `subscribe`. Functions are easier to implement and can be used anywhere in your pipeline, not just channel logic.
:::
### Process directives
Plugins that implement a custom executor will likely need to access {ref}`process directives <process-directives>` that affect the task execution. When an executor receives a task, the process directives can be accessed through that task’s configuration. Custom executors should try to support all process directives that have executor-specific behavior and are relevant to the executor.
Nextflow does not provide the ability to define custom process directives in a plugin. Instead, use the {ref}`process-ext` directive to provide custom process settings to your executor. Use specific names that are not likely to conflict with other plugins or existing pipelines.
For example, a custom executor can use existing process directives and a custom setting through the `ext` directive:
```groovy
class MyExecutor extends Executor {
@Override
TaskHandler createTaskHandler(TaskRun task) {
final cpus = task.config.cpus
final memory = task.config.memory
final myOption = task.config.ext.myOption
println "This task is configured with cpus=${cpus}, memory=${memory}, myOption=${myOption}"
// ...
}
// ...
}
```
(plugins-trace-observers)=
### Trace observers
:::{versionchanged} 25.04.0
The `TraceObserver` interface is now deprecated. Use [TraceObserverV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverV2.groovy) and [TraceObserverFactoryV2](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserverFactoryV2.groovy) instead.
:::
A *trace observer* is an entity that can listen and react to workflow events, such as when a workflow starts, a task is completed, or a file is published. Several components in Nextflow, such as the execution report and DAG visualization, are implemented as trace observers.
Plugins can define custom trace observers that react to workflow events with custom behavior. To implement a trace observer, create a class that implements the `TraceObserver` trait and another class that implements the `TraceObserverFactory` interface. Implement any of the hooks defined in `TraceObserver` and implement the `create()` method in your observer factory. For example:
```groovy
import java.nio.file.Path
import nextflow.processor.TaskHandler
import nextflow.trace.TraceObserver
import nextflow.trace.TraceRecord
class MyObserver implements TraceObserver {
@Override
void onFlowBegin() {
println "Okay, let's begin!"
}
@Override
void onProcessComplete(TaskHandler handler, TraceRecord trace) {
println "I completed a task! It's name is '${handler.task.name}'"
}
@Override
void onProcessCached(TaskHandler handler, TraceRecord trace) {
println "I found a task in the cache! It's name is '${handler.task.name}'"
}
@Override
void onFilePublish(Path destination, Path source) {
println "I published a file! It's located at ${path.toUriString()}"
}
@Override
void onFlowError(TaskHandler handler, TraceRecord trace) {
println "Uh oh, something went wrong..."
}
@Override
void onFlowComplete() {
println 'All done!'
}
}
```
You can then use your trace observer by simply enabling the plugin in your pipeline. In the above example, the observer must also be enabled with a config option:
```nextflow
myplugin.enabled = true
```
See the [`TraceObserver` source code](https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/trace/TraceObserver.groovy) for descriptions of the available workflow events.
(dev-plugins-env-vars)=
## Environment variables
The following environment variables are available to develop and test plugins:
`NXF_PLUGINS_MODE`
: The plugin execution mode. Either `prod` for production or `dev` for development.
`NXF_PLUGINS_DIR`
: The path where the plugin archives are loaded and stored (default: `$NXF_HOME/plugins` in production and `./plugins` in development).
`NXF_PLUGINS_DEFAULT`
: Whether to use the default plugins when no plugins are specified in the Nextflow configuration (default: true).
`NXF_PLUGINS_DEV`
: Comma-separated list of development plugin root directories.
`NXF_PLUGINS_TEST_REPOSITORY`
: :::{versionadded} 23.04.0.
:::
: Comma-separated list of URIs for additional plugin registries or meta files, which will be used in addition to the default registry.
: The URI should refer to a plugin repository JSON file or a specific plugin JSON meta file. In the latter case, it should match the pattern `https://host.name/some/path/<PLUGIN_NAME>-X.Y.Z-meta.json`. For example:
```bash
# custom plugin repository at https://github.com/my-org/plugins
export NXF_PLUGINS_TEST_REPOSITORY="https://raw.githubusercontent.com/my-org/plugins/main/plugins.json"
# custom plugin release
export NXF_PLUGINS_TEST_REPOSITORY="https://github.com/nextflow-io/nf-hello/releases/download/0.3.0/nf-hello-0.3.0-meta.json"
nextflow run main.nf -plugins nf-hello
```
: This variable is useful for testing a plugin release before publishing it to the main registry.

65
nextflow/docs/plugins/plugin-registry.md Normal file
View File

@@ -0,0 +1,65 @@
(plugin-registry-page)=
# Nextflow plugin registry
The [Nextflow plugin registry](https://registry.nextflow.io/) is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking.
Nextflow 25.10 and later can use the plugin registry as a drop-in replacement for the [legacy plugin index](https://github.com/nextflow-io/plugins) hosted on GitHub. See {ref}`migrate-plugin-page` for more information about migrating to the Nextflow plugin registry.
See {ref}`gradle-plugin-publish` for instructions on how to publish plugins to the registry, including the {ref}`README.md requirement <gradle-plugin-readme>`.
(plugin-registry-claim)=
## Claiming a plugin
Ownership of a plugin is required to publish plugins to the Nextflow plugin registry.
To claim ownership of a plugin:
1. Open the [Nextflow plugin registry](https://registry.nextflow.io/) in a browser.
2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address.
:::{note}
If you are logging in for the first time, Seqera will send an authentication link to your email address to complete the login process.
:::
3. Go to the **My plugins** page and select **Claim a plugin**.
4. Enter your unique plugin name or select the plugin you wish to claim in the **Plugin name** field.
5. Enter your organization name in the **Provider** field.
:::{note}
Your organization must match the provider specified when publishing your plugin.
:::
6. Select **Submit Request**.
The plugin will show as **PENDING REVIEW** under **Pending Ownership Requests** until an admin approves the claim. Admin approval is required only once.
(plugin-registry-access-token)=
## Creating an access token
An API access token is required to publish plugins to the Nextflow plugin registry.
To create an API access token:
1. Open the [Nextflow plugin registry](https://registry.nextflow.io/) in a browser.
2. Log in to [Seqera](https://cloud.seqera.io/login) with your GitHub or Google account, or by providing an email address.
:::{note}
If you are logging in for the first time, Seqera will send an authentication link to your email address to complete the login process.
:::
3. Go to the **Access tokens** page.
4. Under **Create New Access Token**, enter a descriptive name for the **Token name** and select the token duration from the **Expiry** drop down.
5. Select **Generate token**.
6. Copy and past token somewhere safe, you won't be able to see it again.
Once you have your token, see {ref}`gradle-plugin-publish` for instructions on how to use it.

57
nextflow/docs/plugins/plugins.md Normal file
View File

@@ -0,0 +1,57 @@
(plugins-page)=
# Overview
## What are plugins
Nextflow plugins are extensions that enhance the functionality of Nextflow. They allow users to add new capabilities and integrate with external services without bloating their pipeline code or modifying Nextflow itself.
There are two types of Nextflow plugins: core plugins and third-party plugins. The main features of each plugin type are described below.
<h3>Core plugins</h3>
Core plugins do not require configuration. The latest versions of core plugins are automatically installed when a Nextflow pipeline requests them. Core plugins include:
* `nf-amazon`: Support for Amazon Web Services.
* `nf-azure`: Support for Microsoft Azure.
* `nf-cloudcache`: Support for the cloud cache.
* `nf-console`: Implementation of the Nextflow [REPL console](https://seqera.io/blog/introducing-nextflow-console/).
* `nf-google`: Support for Google Cloud.
* `nf-tower`: Support for [Seqera Platform](https://seqera.io/platform/).
* `nf-wave`: Support for [Wave containers service](https://seqera.io/wave/).
Specific versions of core plugins can be declared in Nextflow configuration files or by using the `-plugins` option. See {ref}`using-plugins-page` for more information.
:::{note}
The automatic retrieval of core plugins can be disabled by setting `NXF_PLUGINS_DEFAULT=false`. See {ref}`dev-plugins-env-vars` for more information.
:::
<h3>Third-party plugins</h3>
Third-party plugins must be configured via Nextflow configuration files or at runtime. To configure a plugin via configuration files, use the `plugins` block. For example:
```groovy
plugins {
id 'nf-hello@0.5.0'
}
```
To configure plugins at runtime, use the `-plugins` option. For example:
```bash
nextflow run main.nf -plugins nf-hello@0.5.0
```
See {ref}`using-plugins-page` for more information.
## Versioning
Nextflow plugins are free to use any versioning convention. Many plugins, including all core plugins, use [Semantic Versioning](https://semver.org/), which helps developers communicate the kind of changes in a release.
Semantic versions have the form MAJOR.MINOR.PATCH (e.g., 0.5.0):
* MAJOR: Increment for backward-incompatible changes.
* MINOR: Increment for backward-compatible feature additions.
* PATCH: Increment for backward-compatible bug fixes.
Optional pre-release and build metadata can be added (e.g., 1.2.1-alpha+001) as extensions to the base version format.

79
nextflow/docs/plugins/using-plugins.md Normal file
View File

@@ -0,0 +1,79 @@
(using-plugins-page)=
# Using plugins
Nextflow core plugins require no additional configuration. When a pipeline uses a core plugin, Nextflow automatically downloads and uses the latest compatible plugin version. In contrast, third-party plugins must be explicitly declared. When a pipeline uses one or more third-party plugins, Nextflow must be configured to download and use the plugin.
(using-plugins-identifiers)=
## Identifiers
A plugin identifier consists of the plugin name and version, separated by an `@` symbol:
```
nf-hello@0.5.0
```
The plugin version is optional. If it is not specified, Nextflow will download the latest version of the plugin that meets the minimum Nextflow version requirements specified by the plugin.
:::{note}
Plugin versions are required for {ref}`offline usage <using-plugins-offline>`.
:::
:::{versionadded} 25.04.0
:::
The plugin version can be prefixed with `~` to pin the major and minor versions and allow the latest patch release to be used. For example, `nf-amazon@~2.9.0` will resolve to the latest version matching `2.9.x`. When working offline, Nextflow will resolve version ranges against the local plugin cache defined by `NXF_PLUGINS_DIR`.
:::{tip}
It is recommended to pin the major and minor version of each plugin to minimize the risk of breaking changes while allowing patch updates to be used automatically.
:::
(using-plugins-config)=
## Configuration
Plugins can be configured via Nextflow configuration files or at runtime.
To configure a plugin via configuration files, use the `plugins` block. For example:
```nextflow
plugins {
id 'nf-hello@0.5.0'
id 'nf-amazon@2.9.0'
}
```
To configure plugins at runtime, use the `-plugins` option. For example:
```bash
nextflow run main.nf -plugins nf-hello@0.5.0,nf-amazon@2.9.0
```
:::{note}
Plugin declarations in Nextflow configuration files are ignored when specifying plugins via the `-plugins` option.
:::
## Caching
When Nextflow downloads plugins, it caches them in the directory specified by `NXF_PLUGINS_DIR` (`$HOME/.nextflow/plugins` by default).
(using-plugins-offline)=
## Offline usage
When running Nextflow in an offline environment, any required plugins must be downloaded and moved into the offline environment prior to any runs.
To use Nextflow plugins in an offline environment:
1. Install a self-contained version of Nextflow in an environment with an internet connection. See {ref}`install-standalone` for more information.
2. Run `nextflow plugin install <PLUGIN_NAME>@<VERSION>` for each required plugin to download it. Alternatively, run the pipeline once, which will automatically download all plugins required by the pipeline.
3. Copy the `nextflow` binary and `$HOME/.nextflow` directory to the offline environment.
4. Specify each plugin and its version in Nextflow configuration files or at runtime. See {ref}`using-plugins-config` for more information.
:::{warning}
Nextflow will attempt to download newer versions of plugins if their versions are not set. See {ref}`using-plugins-identifiers` for more information.
:::