kevin/ma
1
0
Fork 0
Code Issues 1 Packages Activity

add nextflow d30e48d

Browse Source
This commit is contained in:
Kevin
2026-04-29 23:01:54 +02:00
parent d0b12d668d
commit 97cc9058d3
2840 changed files with 730250 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
nextflow/docs/guides/aws-java-sdk-v2.md Normal file
View File

@@ -0,0 +1,38 @@
(aws-java-sdk-v2-page)=
# AWS Java SDK v2
AWS Java SDK v1 will reach end of life at the end of 2025. Starting in Nextflow 25.10, Nextflow uses AWS Java SDK v2 in the `nf-amazon` plugin.
This migration introduces several breaking changes to the `aws.client` config scope, including new and removed options. This page describes these changes and how they affect your Nextflow configuration.
## New HTTP client
The HTTP client in SDK v2 does not support overriding certain advanced HTTP options. As a result, the following config options are no longer supported:
- `aws.client.protocol`
- `aws.client.signerOverride`
- `aws.client.socketRecvBufferSizeHint`
- `aws.client.socketSendBufferSizeHint`
- `aws.client.userAgent`
## Parallel S3 operations
Nextflow manages S3 transfers, including uploads, downloads, and S3-to-S3 copies, separately from other S3 API calls such as listing a directory or retrieving object metadata.
Use the `aws.client.targetThroughputInGbps` option to control the concurrency of S3 transfers based on the available network bandwidth. This setting defaults to `10`, which allows Nextflow to perform concurrent S3 transfers up to 10 Gbps of network throughput.
Use the `aws.client.maxConnections` config option to control the maximum number of concurrent HTTP connections for all other S3 API calls.
## Multi-part transfers
Nextflow transfers large files to and from S3 as multipart transfers. Use the `aws.client.minimumPartSize` and `aws.client.multipartThreshold` configuration options to control when and how multipart transfers are performed.
Concurrent multipart downloads can consume a large amount of heap memory due to the buffer allocated by each transfer. To avoid out-of-memory errors, the size consumed by these buffers is limited to 400 MB by default. Use the `aws.client.maxDownloadHeapMemory` option to control this value.
The following multipart upload config options are no longer supported:
- `aws.client.uploadChunkSize`
- `aws.client.uploadMaxAttempts`
- `aws.client.uploadMaxThreads`
- `aws.client.uploadRetrySleep`

133
nextflow/docs/guides/gradle-plugin.md Normal file
View File

@@ -0,0 +1,133 @@
(gradle-plugin-page)=
# Using the Nextflow Gradle plugin
Nextflow provides a new `plugin create` command that simplifies the creation of Nextflow plugins. This command leverages the [nf-plugin-template](https://github.com/nextflow-io/nf-plugin-template) project, which uses the [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) to streamline plugin development.
The [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) configures default dependencies needed for Nextflow integration and defines Gradle tasks for building, testing, and publishing Nextflow plugins. The Gradle plugin is versioned and published to the [Gradle Plugin Portal](https://plugins.gradle.org/), allowing developers to manage it like any other dependency. As the plugin ecosystem evolves, the Gradle plugin will enable easier maintenance and adoption of improvements. This page introduces [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) and how to use it.
(gradle-plugin-create)=
## Creating a plugin
:::{versionadded} 25.04.0
:::
The best way to create a plugin with the Nextflow Gradle plugin is to use the `nextflow plugin create` sub-command and create a plugin project based on the [Nextflow plugin template](https://github.com/nextflow-io/nf-plugin-template/). See {ref}`dev-plugins-template` for more information about the Nextflow plugin template.
To create Nextflow plugins with the Gradle plugin:
1. Run `nextflow plugin create`.
2. Follow the prompts to add your plugin name, plugin provider, and project path.
:::{note}
Your plugin provider is usually your organization. This must match the provider specified when claiming your plugin.
:::
3. Develop your plugin extension points. See {ref}`dev-plugins-extension-points` for more information.
4. Develop your tests. See {ref}`gradle-plugin-test` for more information.
5. In the plugin root directory, run `make assemble`.
## Installing a plugin
Plugins can be installed locally without being published.
To install plugins locally:
1. In the plugin root directory, run `make install`.
:::{note}
Running `make install` will add your plugin to your `$HOME/.nextflow/plugins` directory.
:::
2. Run your pipeline:
```bash
nextflow run main.nf -plugins <PLUGIN_NAME>@<VERSION>
```
Replace `<PLUGIN_NAME>@<VERSION>` with your plugin name and version.
:::{note}
Plugins can also be configured via Nextflow configuration files. See {ref}`using-plugins-page` for more information.
:::
(gradle-plugin-test)=
## Testing a plugin
Testing your Nextflow plugin requires unit tests and end-to-end tests.
<h3>Unit tests</h3>
Unit tests are small, focused tests designed to verify the behavior of individual plugin components.
To run unit tests:
1. Develop your unit tests. See [MyObserverTest.groovy](https://github.com/nextflow-io/nf-plugin-template/blob/main/src/test/groovy/acme/plugin/MyObserverTest.groovy) in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example unit test.
2. In the plugin root directory, run `make test`.
<h3>End-to-end tests</h3>
End-to-end tests are comprehensive tests that verify the behavior of an entire plugin as it would be used in Nextflow pipelines.
End-to-end tests should be tailored to the needs of your plugin, but generally take the form of a small Nextflow pipeline. See the `validation` directory in the [plugin template](https://github.com/nextflow-io/nf-plugin-template) for an example end-to-end test.
(gradle-plugin-publish)=
## Publishing a plugin
The Nextflow Gradle plugin allows you to publish plugins to the [Nextflow plugin registry](https://registry.nextflow.io/) from the command line.
(gradle-plugin-readme)=
### README.md requirement
When publishing to the registry, your project must include a `README.md` file in the plugin project root directory. This file will be used as the plugin description in the registry.
**Required sections:**
1. **Summary** - Explain what the plugin does
2. **Get Started** - Setup and configuration instructions
3. **Examples** - Code examples with code blocks
4. **License** - Specify the plugin's license (e.g., Apache 2.0, MIT, GPL)
**Optional sections:**
- **What's new** - Recent changes or new features
- **Breaking changes** - Incompatible changes users should be aware of
**Requirements:**
- Content must be meaningful (no placeholder text like TODO, TBD, Lorem ipsum)
- Content must be in English
### Publishing steps
To publish plugins to the [Nextflow plugin registry](https://registry.nextflow.io/):
1. {ref}`Claim your plugin <plugin-registry-claim>` in the registry.
:::{note}
You can claim a plugin even if it doesn't exist in the registry yet.
:::
2. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory.
3. Add your API key to the file:
```
npr.apiKey=<API_KEY>
```
Replace `<API_KEY>` with your plugin registry API key. See {ref}`plugin-registry-access-token` for more information.
4. Run `make release`.
## Additional resources
For additional Nextflow Gradle plugin configuration options, see the [Nextflow Gradle plugin](https://github.com/nextflow-io/nextflow-plugin-gradle) repository.

117
nextflow/docs/guides/migrate-plugin.md Normal file
View File

@@ -0,0 +1,117 @@
(migrate-plugin-page)=
# Migrating to the Nextflow plugin registry
The [Nextflow plugin registry](https://registry.nextflow.io/) is a central repository for Nextflow plugins. It hosts an index of plugin metadata that supports plugin discovery, accessibility, and version tracking.
The [legacy plugin index](https://github.com/nextflow-io/plugins) is deprecated in favor of the Nextflow plugin registry. Starting with Nextflow 25.10, the plugin registry is the only method for downloading plugins, replacing the legacy plugin index.
This page describes the registry's impact on users and developers, and how to migrate existing plugins.
## Migration impact on plugin users
No immediate actions are required for plugin users. The plugin configuration has not changed.
## Impact on plugin developers
Plugin developers will need to update their plugin to publish to the Nextflow plugin registry instead of the legacy plugin index. The best way to do this is to migrate to the {ref}`Nextflow Gradle plugin<gradle-plugin-page>`, which simplifies the development process and supports publishing to the plugin registry from the command line.
## Migrating to the Nextflow Gradle plugin
To migrate an existing Nextflow plugin to the {ref}`Nextflow Gradle plugin<gradle-plugin-page>`:
1. Remove the following files and folders:
- `buildSrc/`
- `launch.sh`
- `plugins/build.gradle`
2. If your plugin has a `plugins` directory, move the `src` directory to the project root.
:::{note}
Plugin sources should be in `src/main/groovy` or `src/main/java`.
:::
3. Replace the contents of `settings.gradle` with the following:
```groovy
rootProject.name = '<PLUGIN_NAME>'
```
Replace `<PLUGIN_NAME>` with your plugin name.
4. In the project root, create a new `build.gradle` file with the following configuration:
```groovy
// Plugins
plugins {
id 'io.nextflow.nextflow-plugin' version '1.0.0-beta.6'
}
// Dependencies (optional)
dependencies {
<DEPENDENCY>
}
// Plugin version
version = '<PLUGIN_VERSION>'
nextflowPlugin {
// Minimum Nextflow version
nextflowVersion = '<MINIMUM_NEXTFLOW_VERSION>'
// Plugin metadata
provider = '<PROVIDER>'
className = '<CLASS_NAME>'
extensionPoints = [
'<EXTENSION_POINT>'
]
}
```
Replace the following:
- `<DEPENDENCY>`: (Optional) Your plugins dependency libraries. For example, `commons-io:commons-io:2.18.0`.
- `<PLUGIN_VERSION>:` Your plugin version. For example, `0.5.0`.
- `<MINIMUM_NEXTFLOW_VERSION>`: The minimum Nextflow version required to run your plugin. For example, `25.04.0`.
- `<PROVIDER>`: Your name or organization. For example, `acme`.
- `<CLASS_NAME>`: Your plugin class name. For example, `acme.plugin.MyPlugin`.
- `<EXTENSION_POINT>`: Your extension point identifiers that the plugin will implement or expose. For example, `acme.plugin.MyFactory`.
5. Replace the contents of `Makefile` with the following:
```Makefile
# Build the plugin
assemble:
./gradlew assemble
clean:
rm -rf .nextflow*
rm -rf work
rm -rf build
./gradlew clean
# Run plugin unit tests
test:
./gradlew test
# Install the plugin into local nextflow plugins dir
install:
./gradlew install
# Publish the plugin
release:
./gradlew releasePlugin
```
6. Update `README.md` with information about the structure of your plugin.
7. In the plugin root directory, run `make assemble`.
Alternatively, use the `nextflow plugin create` command to re-create your plugin with the plugin template and add your existing plugin code. See {ref}`dev-plugins-template` for more information about the plugin template.
## Publishing to the Nextflow plugin registry
The Nextflow Gradle plugin supports publishing plugins from the command line. See {ref}`gradle-plugin-publish` for more information.
The legacy plugin index is **closed to new pull requests**. It will be kept up-to-date with the plugin registry and will remain available indefinitely to ensure continued support for older versions of Nextflow.

86
nextflow/docs/guides/updating-spot-retries.md Normal file
View File

@@ -0,0 +1,86 @@
(spot-retries-page)=
# Spot Instance failures and retries
This page describes changes in how Nextflow handles Spot Instance failures and retries on AWS and Google Cloud, the impact of those changes, and how to configure spot retry behavior for your pipelines. These changes apply to Nextflow 24.10 and later.
## Retry behavior
Up to version 24.10, Nextflow would silently retry Spot Instance failures up to `5` times when using AWS Batch or Google Batch. These retries were controlled by cloud-specific configuration parameters (e.g., `aws.batch.maxSpotAttempts`) and happened in cloud infrastructure without explicit visibility to Nextflow.
<h3>Before Nextflow 24.10</h3>
By default, Nextflow would instruct AWS and Google to automatically retry jobs lost to Spot reclamation up to `5` times. Retries were handled by the cloud provider _within_ a Nextflow task. It was often unclear that tasks were restarted as there was no explicit message. Task runtimes and associated cloud costs were increased because they included the runtime of the reclaimed and retried tasks. Due to the high likelihood of reclamation before completion, long-running tasks running on Spot Instances frequently required retries, leading to inefficient allocation of resources and higher costs.
<h3>After Nextflow 24.10</h3>
The default Spot reclamation retry setting changed to `0` on AWS and Google. By default, no _internal_ retries are attempted on these platforms. Spot reclamations lead to an immediate failure, exposed to Nextflow in the same way as other generic failures (for example, returning `exit code 1` on AWS). Nextflow treats these failures like any other job failure, unless a retry strategy is configured.
## Impact on existing workflows
If you rely on silent Spot retries (the previous default behavior), you may now see more tasks fail with the following characteristics:
- **AWS**: Generic failure with `exit code 1`. You may see messages indicating the host machine was terminated.
- **Google**: Spot reclamation typically produces a specific code, but is now surfaced as a recognizable task failure in Nextflow logs.
Since the default for Spot retries is now `0`, you must actively enable a retry strategy if you want Nextflow to handle reclaimed Spot Instances automatically.
## Possible actions
There are four possible actions.
### Do nothing
If you do not configure anything, you will observe more pipeline failures when Spot Instances are reclaimed. This approach provides clearer visibility into failures. Failed tasks can be re-run with the `-resume` option. However, frequent task reclamation may lead to a higher failure rate and each retry requires manual intervention.
:::{note}
If you resume the pipeline using the resume option, it will pick up at the point the pipeline was interrupted and start with a retry of that task.
:::
### Re-enable Spot retries
You can re-enable Spot retries for each provider in your Nextflow config:
```groovy
aws.batch.maxSpotAttempts = 5
google.batch.maxSpotAttempts = 5
```
The above example sets the maximum number of Spot retries to `5` for both AWS and Google.
### Make Spot failures visible and retry them
You can set `maxRetries` to enable Nextflow-level retries for any failure:
```groovy
process.maxRetries = 5
```
The above example sets retries to `5` for any failures across all providers.
### Use Fusion Snapshots
*This option is only available for AWS Batch.*
If you have long-running tasks where progress lost due to Spot reclamation is costly, consider [Fusion Snapshots](https://docs.seqera.io/fusion/guide/snapshots) (if supported by your environment). Fusion Snapshots allow you to resume a partially completed task on a new machine if a Spot Instance is reclaimed, thereby reducing wasted compute time.
Key features of Fusion Snapshots:
- Jobs do not need to start from scratch after reclamation.
- Especially useful for tasks that take many hours or days to complete.
- May significantly reduce costs and run times in high-reclamation environments.
See [Fusion Snapshots for AWS Batch](https://docs.seqera.io/fusion/guide/snapshots) for more information.
## Best practices
Best practices for Spot Instance failures and retries:
- **Evaluate job duration**: If your tasks are very long (multiple hours or more), Spot Instances can cause repeated interruptions. Consider using on-demand instances or Fusion Snapshots.
- **Set sensible retry limits**: If you enable Spot retries, choose a retry count that balances the cost savings of Spot usage against the overhead of restarting tasks.
- **Monitor logs and exit codes**: Failures due to Spot reclamation will now appear in Nextflow logs. Monitor failures and fine-tune your strategy.
- **Consider partial usage of Spot**: Some workflows may mix on-demand instances for critical or long tasks and Spot Instances for shorter, less critical tasks. This can optimize cost while minimizing wasted compute time.