kevin/ma
1
0
Fork 0
Code Issues 1 Packages Activity

add nextflow d30e48d

Browse Source
This commit is contained in:
Kevin
2026-04-29 23:01:54 +02:00
parent d0b12d668d
commit 97cc9058d3
2840 changed files with 730250 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
nextflow/specs/251117-module-system/checklists/requirements.md Normal file
View File

@@ -0,0 +1,39 @@
# Specification Quality Checklist: Nextflow Module System Client
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2026-01-15
**Feature**: [spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- Specification is complete and ready for `/speckit.plan`
- All 8 user stories have clear acceptance scenarios
- 29 functional requirements defined across 6 categories
- 8 success criteria defined with measurable outcomes
- Edge cases documented for error handling scenarios
- Registry backend is explicitly out of scope (assumed implemented)

388
nextflow/specs/251117-module-system/contracts/registry-api.yaml Normal file
View File

@@ -0,0 +1,388 @@
openapi: 3.0.3
info:
title: Nextflow Module Registry API
description: |
API specification for the Nextflow Module Registry.
This documents the endpoints that the Nextflow module system client consumes.
The registry backend is assumed to be already implemented at registry.nextflow.io.
version: 1.0.0
contact:
name: Nextflow Team
url: https://nextflow.io
servers:
- url: https://registry.nextflow.io/api
description: Production registry
security:
- BearerAuth: []
paths:
/modules:
get:
operationId: searchModules
summary: Search modules
description: Search for modules by query text (semantic search across name, description, keywords)
tags:
- Modules
parameters:
- name: query
in: query
required: true
description: Search query text
schema:
type: string
example: "alignment"
- name: limit
in: query
required: false
description: Maximum number of results
schema:
type: integer
default: 10
maximum: 100
responses:
'200':
description: Search results
content:
application/json:
schema:
type: object
properties:
modules:
type: array
items:
$ref: '#/components/schemas/ModuleSummary'
total:
type: integer
description: Total matching modules
'400':
description: Invalid query parameters
/modules/{name}:
get:
operationId: getModule
summary: Get module details
description: Get module metadata including latest release information
tags:
- Modules
parameters:
- name: name
in: path
required: true
description: Module name including scope (e.g., nf-core/fastqc)
schema:
type: string
example: "nf-core/fastqc"
responses:
'200':
description: Module details
content:
application/json:
schema:
$ref: '#/components/schemas/ModuleDetails'
'404':
description: Module not found
post:
operationId: publishModule
summary: Publish module version
description: Upload and publish a new module version (requires authentication)
tags:
- Modules
security:
- BearerAuth: []
parameters:
- name: name
in: path
required: true
description: Module name including scope
schema:
type: string
requestBody:
required: true
content:
multipart/form-data:
schema:
type: object
required:
- bundle
properties:
bundle:
type: string
format: binary
description: Module bundle (tar.gz archive)
tags:
type: array
items:
type: string
description: Additional tags for discoverability
responses:
'201':
description: Module published successfully
content:
application/json:
schema:
$ref: '#/components/schemas/PublishResult'
'400':
description: Invalid module bundle or manifest
content:
application/json:
schema:
$ref: '#/components/schemas/ValidationError'
'401':
description: Authentication required
'403':
description: Insufficient permissions for scope
/modules/{name}/releases:
get:
operationId: listReleases
summary: List module releases
description: Get all available versions of a module
tags:
- Modules
parameters:
- name: name
in: path
required: true
schema:
type: string
responses:
'200':
description: List of releases
content:
application/json:
schema:
type: object
properties:
releases:
type: array
items:
$ref: '#/components/schemas/ReleaseInfo'
'404':
description: Module not found
/modules/{name}/{version}:
get:
operationId: getRelease
summary: Get specific release
description: Get metadata for a specific module version
tags:
- Modules
parameters:
- name: name
in: path
required: true
schema:
type: string
- name: version
in: path
required: true
description: Semantic version (e.g., 1.0.0)
schema:
type: string
example: "1.0.0"
responses:
'200':
description: Release details
content:
application/json:
schema:
$ref: '#/components/schemas/ReleaseInfo'
'404':
description: Module or version not found
/modules/{name}/{version}/download:
get:
operationId: downloadModule
summary: Download module bundle
description: Download the module source archive for a specific version
tags:
- Modules
parameters:
- name: name
in: path
required: true
schema:
type: string
- name: version
in: path
required: true
schema:
type: string
responses:
'200':
description: Module bundle
headers:
X-Checksum:
description: SHA-256 checksum of the bundle
schema:
type: string
example: "sha256:abc123..."
Content-Disposition:
description: Suggested filename
schema:
type: string
example: "attachment; filename=nf-core-fastqc-1.0.0.tar.gz"
content:
application/gzip:
schema:
type: string
format: binary
'404':
description: Module or version not found
components:
securitySchemes:
BearerAuth:
type: http
scheme: bearer
description: |
Authentication token. Can be provided via:
- NXF_REGISTRY_TOKEN environment variable
- registry.auth config block
schemas:
ModuleSummary:
type: object
required:
- name
- latestVersion
- description
properties:
name:
type: string
description: Full module name with scope
example: "nf-core/fastqc"
latestVersion:
type: string
description: Latest available version
example: "1.2.0"
description:
type: string
description: Short module description
example: "Run FastQC on sequenced reads"
downloadCount:
type: integer
description: Total download count
example: 15420
keywords:
type: array
items:
type: string
example: ["quality control", "fastq"]
ModuleDetails:
type: object
required:
- name
- latestVersion
- description
properties:
name:
type: string
example: "nf-core/fastqc"
latestVersion:
type: string
example: "1.2.0"
description:
type: string
authors:
type: array
items:
type: string
example: ["@drpatelh"]
maintainers:
type: array
items:
type: string
license:
type: string
example: "MIT"
keywords:
type: array
items:
type: string
downloadCount:
type: integer
createdAt:
type: string
format: date-time
updatedAt:
type: string
format: date-time
latestRelease:
$ref: '#/components/schemas/ReleaseInfo'
ReleaseInfo:
type: object
required:
- version
- checksum
- publishedAt
properties:
version:
type: string
description: Semantic version
example: "1.2.0"
checksum:
type: string
description: SHA-256 checksum of bundle
example: "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
publishedAt:
type: string
format: date-time
size:
type: integer
description: Bundle size in bytes
example: 45678
requires:
type: object
properties:
nextflow:
type: string
example: ">=24.04.0"
plugins:
type: array
items:
type: string
modules:
type: array
items:
type: string
PublishResult:
type: object
properties:
name:
type: string
version:
type: string
checksum:
type: string
publishedAt:
type: string
format: date-time
downloadUrl:
type: string
format: uri
ValidationError:
type: object
properties:
code:
type: string
enum:
- INVALID_MANIFEST
- MISSING_MAIN_NF
- MISSING_README
- INVALID_VERSION
- BUNDLE_TOO_LARGE
- DUPLICATE_VERSION
message:
type: string
details:
type: array
items:
type: string

272
nextflow/specs/251117-module-system/data-model.md Normal file
View File

@@ -0,0 +1,272 @@
# Data Model: Nextflow Module System Client
**Date**: 2026-01-19
**Feature**: 251117-module-system
**Last Updated**: 2026-02-27 (reflects final implementation)
## Overview
This document defines the data entities, their attributes, relationships, and state transitions for the Nextflow module system client implementation.
---
## Entity Definitions
### 1. ModuleReference
Represents a reference to a module in DSL `include` statements.
```groovy
@CompileStatic
class ModuleReference {
String scope // e.g., "nf-core"
String name // e.g., "fastqc"
String fullName // e.g., "@nf-core/fastqc"
static ModuleReference parse(String source) {
// Parses "@scope/name" format
}
boolean isRegistryModule() {
return fullName.startsWith('@')
}
}
```
**Validation Rules**:
- `scope`: lowercase alphanumeric with hyphens, pattern `[a-z0-9][a-z0-9-]*`
- `name`: lowercase alphanumeric with underscores/hyphens, pattern `[a-z][a-z0-9_-]*`
- `fullName`: must match `^@[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$`
---
### 2. ModuleSpec
Parsed representation of `meta.yaml` file. Class: `nextflow.module.ModuleSpec`.
```groovy
@CompileStatic
class ModuleSpec {
String name // e.g., "nf-core/fastqc" (without @)
String version // e.g., "1.0.0"
String description // Module description
List<String> keywords // Discovery keywords
List<String> authors // GitHub handles
String license // SPDX identifier
Map<String, String> requires // dependency -> version constraint
static ModuleSpec load(Path metaYamlPath) { ... }
List<String> validate() { ... } // Returns list of validation errors
boolean isValid() { ... }
}
```
**Validation Rules**:
- `name`: Must match `scope/name` or `scope/path/to/name` pattern
- `version`: Must be valid SemVer (`MAJOR.MINOR.PATCH[-prerelease]`)
- `description`, `license`: Required fields (validate() reports missing)
**Note**: Tool/argument definitions were removed from the ADR and are not part of `ModuleSpec`.
---
### 3. InstalledModule
Represents a module in the local `modules/` directory.
```groovy
@CompileStatic
class InstalledModule {
ModuleReference reference
Path directory // e.g., /project/modules/@nf-core/fastqc
Path mainFile // e.g., /project/modules/@nf-core/fastqc/main.nf
Path manifestFile // e.g., /project/modules/@nf-core/fastqc/meta.yaml
Path checksumFile // e.g., /project/modules/@nf-core/fastqc/.checksum
String installedVersion
String expectedChecksum
ModuleIntegrity getIntegrity() {
// Compute and compare checksum
}
}
enum ModuleIntegrity {
VALID, // Checksum matches
MODIFIED, // Checksum mismatch (local changes)
MISSING_CHECKSUM, // No .checksum file
CORRUPTED // Missing required files
}
```
**State Transitions**:
```
[NOT_INSTALLED] --install--> [VALID]
[VALID] --user edits--> [MODIFIED]
[MODIFIED] --install -force--> [VALID]
[VALID] --version change in config--> [VALID] (replaced)
[MODIFIED] --version change in config--> [MODIFIED] (blocked, warn)
```
---
### 4. ModulesConfig and RegistryConfig
Modules configuration loaded from `nextflow_spec.json` ( or the `modules {}` block in `nextflow.config` as alternative). Registry settings from the `registry {}` block in `nextflow.config`.
```groovy
@ScopeName("modules")
@CompileStatic
class ModulesConfig implements ConfigScope {
Map<String, String> modules = [:] // module fullName -> version
String getVersion(String moduleName) { ... }
boolean hasVersion(String moduleName) { ... }
}
@ScopeName("registry")
@CompileStatic
class RegistryConfig implements ConfigScope {
static final String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api'
Collection<String> url // Registry URL(s) in priority order
String apiKey // API key (falls back to NXF_REGISTRY_TOKEN env var)
String getUrl() // Returns primary (first) URL
Collection<String> getAllUrls()
String getApiKey() // Returns apiKey or NXF_REGISTRY_TOKEN
}
```
**Config Syntax**:
```nextflow
// nextflow_spec.json (current approach)
{
"modules": {
"@nf-core/fastqc": "1.0.0",
"@nf-core/bwa-align": "1.2.0"
}
}
// nextflow.config (alternative not currently used)
modules {
'@nf-core/fastqc' = '1.0.0'
'@nf-core/bwa-align' = '1.2.0'
}
registry {
url = [
'https://private.registry.myorg.com',
'https://registry.nextflow.io/api'
]
apiKey = '${MYORG_TOKEN}' // Only applied to the primary registry
}
```
---
### 5. DefaultRemoteModuleResolver (SPI)
Bridges the DSL parser to the module resolution runtime. Class: `nextflow.module.DefaultRemoteModuleResolver`.
```groovy
// Implements: nextflow.module.spi.RemoteModuleResolver (nf-lang)
class DefaultRemoteModuleResolver implements RemoteModuleResolver {
int getPriority() { return 0 } // Can be overridden by plugins with higher priority
Path resolve(String moduleName, Path baseDir) {
// 1. Parse ModuleReference from "@scope/name"
// 2. Read version constraints from nextflow_spec.json / ModulesConfig
// 3. Call ModuleResolver.installModule(reference, version, autoInstall=true)
// 4. Return path to modules/@scope/name/main.nf
}
}
```
The SPI is loaded via Java `ServiceLoader` by `RemoteModuleResolverProvider` (in `nf-lang`), which selects the highest-priority implementation available.
---
### 6. PipelineSpec
Reads and writes `nextflow_spec.json` in the project root. Class: `nextflow.pipeline.PipelineSpec`.
```groovy
class PipelineSpec {
PipelineSpec(Path baseDir)
Map<String, String> getModules()
void addModuleEntry(String name, String version)
boolean removeModuleEntry(String name)
}
```
---
### 6. ModuleResolutionResult
Result of module resolution process.
```groovy
@CompileStatic
class ModuleResolutionResult {
ModuleReference reference
Path resolvedPath // Absolute path to main.nf
ResolutionAction action
String message // Warning/info message if any
}
enum ResolutionAction {
USE_LOCAL, // Used existing local module
DOWNLOADED, // Downloaded from registry
REPLACED, // Replaced local with different version
BLOCKED_MODIFIED, // Local modified, not replaced (warning issued)
FAILED // Resolution failed (error)
}
```
---
## Relationships
```
PipelineSpec (1) -----> (*) ModuleReference (nextflow_spec.json)
ModulesConfig (1) -----> (*) ModuleReference (nextflow.config alternative)
RegistryConfig (1) -----> (*) Registry URLs
ModuleReference (1) -----> (0..1) InstalledModule
|
v (via registry)
ModuleSpec (1) <----- InstalledModule (from meta.yaml)
```
---
## Storage Layout
```
project-root/
├── nextflow.config # registry{} block; optional modules{} block
├── nextflow_spec.json # auto-managed module version pins
├── main.nf # include { X } from '@scope/name'
└── modules/
└── @scope/
└── name/
├── .checksum # SHA-256 from registry (download integrity)
├── main.nf # Entry point (required)
├── meta.yaml # Manifest (required for publishing)
├── README.md # Documentation (required for publishing)
└── [other files] # Supporting files
```
---
## Validation Summary
| Entity | Field | Validation |
|--------|-------|------------|
| ModuleReference | fullName | Pattern: `^@[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$` |
| ModuleSpec | name | Pattern: `scope/name` or `scope/path/to/name` |
| ModuleSpec | version | SemVer: `MAJOR.MINOR.PATCH[-prerelease]` |
| ModuleSpec | description, license | Required (non-empty) |
| InstalledModule | directory | Must contain main.nf |
| ModulesConfig | modules keys | Must be valid module fullName |
| RegistryConfig | url | Valid HTTPS URL |

137
nextflow/specs/251117-module-system/plan.md Normal file
View File

@@ -0,0 +1,137 @@
# Implementation Plan: Nextflow Module System Client
**Branch**: `251117-module-system` | **Date**: 2026-01-19 | **Spec**: [spec.md](spec.md)
**Input**: Feature specification from `/specs/251117-module-system/spec.md`
## Summary
Implement client-side module system for Nextflow enabling pipeline developers to include remote modules from the Nextflow registry using `@scope/name` syntax, manage versions via `nextflow.config`, configure module parameters via `meta.yaml`, and use CLI commands (install, search, list, remove, publish, run). Implementation extends existing DSL parser, config parser, and follows plugin system patterns for registry communication and authentication.
## Technical Context
**Language/Version**: Groovy 4.0.29 (targeting Java 17 runtime, Java 21 toolchain for development)
**Primary Dependencies**:
- Existing Nextflow DSL parser (nf-lang module, ANTLR)
- Existing config parser (ConfigBuilder, ConfigParser)
- Existing HTTP client (HxClient from io.seqera.http)
- Existing plugin authentication infrastructure
- Existing npr-api (registry data models and schema validation)
**Storage**: Local filesystem (`modules/@scope/name/` per-project, `.checksum` files)
**Testing**: Spock Framework for unit tests, integration tests in `tests/` directory
**Target Platform**: JVM 17+ (same as Nextflow core)
**Project Type**: Multi-module Gradle project extension (core modules + CLI)
**Performance Goals**: Module resolution adds <2 seconds to workflow startup when cached locally (SC-002)
**Constraints**:
- Module bundle size limit: 1MB uncompressed (enforced by registry)
- Backward compatibility: Must not break existing `include` statements
- Offline operation: Must work with locally cached modules
**Scale/Scope**: Ecosystem-wide module distribution; typical project: 5-20 modules
## Constitution Check
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
| Principle | Status | Evidence |
|-----------|--------|----------|
| I. Modular Architecture | PASS | Module system client belongs in `modules/nextflow` (core CLI) with potential shared utilities in `nf-commons` |
| II. Test-Driven Quality | PASS | Unit tests (Spock), integration tests planned, smoke test support |
| III. Dataflow Programming Model | PASS | Modules are process definitions; include resolution at parse time preserves dataflow semantics |
| IV. Apache 2.0 License | PASS | All new code will include Apache 2.0 headers |
| V. DCO Sign-off | PASS | All commits will use `git commit -s` |
| VI. Semantic Versioning | PASS | Modules use SemVer; plugin-compatible version constraint syntax |
| VII. Groovy Idioms | PASS | Follow existing patterns from CmdPlugin, ConfigBuilder, HttpPluginRepository |
**Gate Status**: PASS - No violations requiring justification
## Project Structure
### Documentation (this feature)
```text
specs/251117-module-system/
├── plan.md # This file
├── spec.md # Feature specification
├── research.md # Phase 0 output
├── data-model.md # Phase 1 output
├── quickstart.md # Phase 1 output
├── contracts/ # Phase 1 output (API contracts)
└── tasks.md # Phase 2 output (from /speckit.tasks)
```
### Source Code (repository root)
```text
modules/nextflow/src/main/groovy/nextflow/
├── cli/
│ ├── CmdModule.groovy # Main module command (uses JCommander)
│ └── module/
│ ├── ModuleInstall.groovy # Install subcommand (extends CmdBase)
│ ├── ModuleRun.groovy # Run subcommand (extends CmdRun)
│ ├── ModuleList.groovy # List subcommand (extends CmdBase)
│ ├── ModuleRemove.groovy # Remove subcommand (extends CmdBase)
│ ├── ModuleSearch.groovy # Search subcommand (extends CmdBase)
│ ├── ModuleInfo.groovy # Info subcommand (extends CmdBase)
│ └── ModulePublish.groovy # Publish subcommand (extends CmdBase)
├── config/
│ ├── ModulesConfig.groovy # modules{} config scope
│ └── RegistryConfig.groovy # registry{} config scope (fields: url, apiKey)
├── module/
│ ├── ModuleReference.groovy # @scope/name parser
│ ├── ModuleResolver.groovy # Core resolution logic (version/integrity/install)
│ ├── ModuleStorage.groovy # Local filesystem operations
│ ├── ModuleRegistryClient.groovy # HTTP registry client
│ ├── ModuleChecksum.groovy # SHA-256 integrity verification
│ ├── ModuleSpec.groovy # Module manifest (meta.yaml) entity
│ ├── InstalledModule.groovy # Installed module entity
│ └── DefaultRemoteModuleResolver.groovy # SPI impl: bridges DSL parser → ModuleResolver
└── pipeline/
└── PipelineSpec.groovy # nextflow_spec.json read/write
modules/nf-lang/src/main/java/nextflow/script/
└── control/ResolveIncludeVisitor.java # MODIFIED: Delegates @scope/name to SPI resolver
modules/nf-lang/src/main/java/nextflow/module/spi/
├── RemoteModuleResolver.java # SPI interface (extensible by plugins)
├── RemoteModuleResolverProvider.java # ServiceLoader wrapper (singleton)
└── FallbackRemoteModuleResolver.java # Error fallback when no impl found
modules/nextflow/src/test/groovy/nextflow/
├── cli/module/
│ ├── ModuleInstallTest.groovy
│ ├── ModuleRunTest.groovy
│ └── [other subcommand tests]
└── module/
├── ModuleResolverTest.groovy
├── ModuleStorageTest.groovy
└── [other module tests]
tests/modules/
├── install-module.nf # Integration tests
├── run-module.nf
└── [other integration tests]
```
**Structure Decision**: Implementation extends existing Nextflow core modules following modular architecture. New code in `modules/nextflow` for CLI and core logic. DSL parser extension in `modules/nf-lang` via SPI. No new plugins required.
## Architecture Notes
### Remote Module Inclusion — SPI Pattern
The DSL parser (`ResolveIncludeVisitor`) detects the `@` prefix in `include` statements and delegates resolution to a `RemoteModuleResolver` SPI loaded via Java `ServiceLoader`. This keeps `nf-lang` decoupled from the runtime module resolution logic:
```
include { X } from '@nf-core/fastqc'
↓
ResolveIncludeVisitor (nf-lang)
source.startsWith("@") → RemoteModuleResolverProvider.getInstance().resolve(...)
↓
DefaultRemoteModuleResolver (nextflow module)
auto-installs via ModuleResolver if missing → returns Path to main.nf
```
The `RemoteModuleResolver` interface in `nf-lang` can be overridden by plugins with a higher priority value.
## Complexity Tracking
No constitution violations requiring justification.

274
nextflow/specs/251117-module-system/quickstart.md Normal file
View File

@@ -0,0 +1,274 @@
# Quickstart: Nextflow Module System
This guide covers the essential workflows for using the Nextflow module system.
## Prerequisites
- Nextflow 25.x or later (with module system support)
- Network connectivity for initial module downloads
- Optional: `NXF_REGISTRY_TOKEN` for publishing
---
## 1. Install and Use a Module
### Install a module
```bash
# Install latest version
nextflow module install nf-core/fastqc
# Install specific version
nextflow module install nf-core/fastqc -version 1.0.0
```
This downloads the module to `modules/@nf-core/fastqc/` and updates `nextflow_spec.json` with the installed version.
### Use in your workflow
```groovy
// main.nf
include { FASTQC } from '@nf-core/fastqc'
workflow {
reads = Channel.fromFilePairs('data/*_{1,2}.fastq.gz')
FASTQC(reads)
}
```
### Run your workflow
```bash
nextflow run main.nf
```
---
## 2. Run a Module Directly
Execute a module without writing a wrapper workflow:
```bash
# Basic usage
nextflow module run nf-core/fastqc --input 'data/*.fastq.gz'
# Run specific version
nextflow module run nf-core/fastqc --input 'data/*.fastq.gz' -version 1.0.0
# With Nextflow options
nextflow module run nf-core/salmon \
--reads reads.fq \
--index salmon_index \
-profile docker \
-resume
```
## 3. View Module Information
```bash
# Show module metadata and a generated usage template
nextflow module info nf-core/fastqc
# Show a specific version
nextflow module info nf-core/fastqc -version 1.0.0
# JSON output for scripting
nextflow module info nf-core/fastqc -json
```
---
## 4. Manage Module Versions
### Version tracking
Module versions are automatically recorded in `nextflow_spec.json` by `nextflow module install`. You can also pin versions manually:
```json
// nextflow_spec.json
{
"modules": {
"@nf-core/fastqc": "1.0.0",
"@nf-core/bwa-align": "1.2.0"
}
}
```
Alternatively, declare versions in `nextflow.config` (not currently used):
```nextflow
modules {
'@nf-core/fastqc' = '1.0.0'
'@nf-core/bwa-align' = '1.2.0'
}
```
### Check module status
```bash
# List all modules
nextflow module list
# Output:
# MODULE CONFIGURED INSTALLED LATEST STATUS
# @nf-core/fastqc 1.0.0 1.0.0 1.2.0 outdated
# @nf-core/bwa-align 1.2.0 1.2.0 1.2.0 up-to-date
# @nf-core/samtools 2.1.0 - 2.1.0 missing
```
### Update a module
Change the version in `nextflow_spec.json` (or `nextflow.config`), then run your workflow. Nextflow automatically downloads the new version.
---
## 5. Search for Modules
```bash
# Search by keyword
nextflow module search alignment
# Limit results
nextflow module search "quality control" -limit 5
# JSON output for scripting
nextflow module search bwa -json
```
---
## 6. Work with Private Registries
### Configure authentication
```nextflow
// nextflow.config
registry {
// Multiple registries (tried in order)
url = [
'https://private.registry.myorg.com',
'https://registry.nextflow.io/api'
]
apiKey = 'MYORG_TOKEN' // Applied to the primary (first) registry only
}
```
### Or use environment variable
```bash
export NXF_REGISTRY_TOKEN=your-token-here
nextflow module install nf-core/fastqc
```
---
## 7. Publish a Module
### Prepare your module
```
my-module/
├── main.nf # Required: entry point
├── meta.yaml # Required for registry
├── README.md # Required for registry
└── tests/ # Recommended
```
### Validate before publishing
```bash
nextflow module publish myorg/my-module -dry-run
```
### Publish to registry
```bash
export NXF_REGISTRY_TOKEN=your-token
nextflow module publish myorg/my-module
```
---
## 8. Handle Local Modifications
If you modify a module locally (for debugging), Nextflow protects your changes:
```bash
# This warns and does NOT override your changes
nextflow module install nf-core/fastqc -version 1.1.0
# Warning: Module @nf-core/fastqc has local modifications. Use -force to override.
# Force replacement if needed
nextflow module install nf-core/fastqc -version 1.1.0 -force
```
---
## 9. Remove a Module
```bash
# Remove module and config entry
nextflow module remove nf-core/fastqc
# Keep config entry (just delete local files)
nextflow module remove nf-core/fastqc -keep-config
# Keep local files (just remove from config)
nextflow module remove nf-core/fastqc -keep-files
```
---
## Common Patterns
### Offline operation
Modules are cached locally in `modules/`. Once installed, workflows run without network access.
### Git integration
The `modules/` directory is intended to be committed to your git repository:
```bash
git add modules/
git commit -m "Add module dependencies"
```
---
## Troubleshooting
### Module not found
```bash
# Check if module exists in registry
nextflow module search exact-module-name
# Verify spelling and scope
# Correct: @nf-core/fastqc
# Wrong: @nfcore/fastqc, nf-core/fastqc (without @)
```
### Authentication errors
```bash
# Verify token is set
echo $NXF_REGISTRY_TOKEN
# Check registry config
grep -A5 'registry' nextflow.config
```
### Version conflicts
If two modules require incompatible versions of a dependency:
- Nextflow selects the highest compatible version automatically
- If no compatible version exists, an error lists the conflicts
### Checksum warnings
```
Warning: Module @nf-core/fastqc has local modifications
```
This means the local module content differs from the registry version. Your changes are preserved. Use `-force` only if you want to discard local changes.

352
nextflow/specs/251117-module-system/research.md Normal file
View File

@@ -0,0 +1,352 @@
# Research: Nextflow Module System Client
**Date**: 2026-01-19
**Feature**: 251117-module-system
## Overview
This document captures technical research and decisions for implementing the Nextflow module system client. All NEEDS CLARIFICATION items from Technical Context have been resolved through codebase exploration.
---
## 1. CLI Command Structure
**Research Question**: How should `nextflow module` CLI commands be implemented?
**Decision**: JCommander native subcommands — each subcommand extends `CmdBase` directly; no trait needed
**Rationale**:
- JCommander's subcommand support handles parameter parsing automatically per subcommand
- Each subcommand (install, run, list, remove, search, info, publish) is a separate class extending CmdBase
- `ModuleRun` extends `CmdRun` to reuse pipeline execution logic (PR #6381)
- No custom `ModuleSubCmd` trait needed; cleaner architecture
- `CmdModule` is registered in `Launcher` alongside all other top-level commands
**Implemented Pattern**:
```groovy
@Parameters(commandDescription = "Manage Nextflow modules")
class CmdModule extends CmdBase implements UsageAware {
static final List<CmdBase> commands = []
static {
commands << new ModuleInstall() // extends CmdBase
commands << new ModuleRun() // extends CmdRun
commands << new ModuleList() // extends CmdBase
commands << new ModuleRemove() // extends CmdBase
commands << new ModuleSearch() // extends CmdBase
commands << new ModuleInfo() // extends CmdBase
commands << new ModulePublish() // extends CmdBase
}
void run() {
final jc = commander() // JCommander with all subcommands registered
jc.parse(args as String[])
final subcommand = jc.getCommands().get(jc.getParsedCommand()).getObjects()[0]
subcommand.run()
}
}
```
**Alternatives Considered**:
- CmdFs trait pattern: Considered initially; replaced by JCommander native subcommands — simpler and avoids custom parsing
- Separate top-level Cmd classes (CmdModuleInstall, etc.): Rejected — too many entry points
- Plugin-based CLI extension: Rejected — module system is core functionality, not optional
---
## 2. DSL Parser Extension for @scope/name
**Research Question**: How to extend `include` statement parsing for registry modules?
**Decision**: Extend `ResolveIncludeVisitor` to detect `@` prefix and delegate to a `RemoteModuleResolver` SPI loaded via Java `ServiceLoader`
**Rationale**:
- Keeps `nf-lang` decoupled from runtime module resolution (`nf-lang` has no dependency on `nextflow` module)
- SPI pattern allows plugins or custom implementations to override the default resolver
- Detection: `source.startsWith('@')` distinguishes registry vs local paths — preserves existing include behavior
- Resolution at parse time (after plugin resolution) per ADR
**Implemented Architecture**:
```
include { X } from '@scope/name'
↓
ResolveIncludeVisitor.visitInclude() [nf-lang]
source.startsWith("@") → RemoteModuleResolverProvider.getInstance().resolve(source, baseDir)
↓
RemoteModuleResolverProvider [nf-lang]
Java ServiceLoader discovers implementations; picks highest priority
↓
DefaultRemoteModuleResolver [nextflow module]
Calls ModuleResolver.installModule(reference, version, autoInstall=true)
Returns Path to modules/@scope/name/main.nf
```
**Key Files**:
- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java` — SPI interface
- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java` — ServiceLoader singleton
- `modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java` — error fallback
- `modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java` — MODIFIED
- `modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy` — default impl
**Alternatives Considered**:
- New ANTLR grammar token for `@`: Rejected — unnecessary parser complexity
- Direct dependency from nf-lang to nextflow module: Rejected — circular dependency risk; SPI decouples cleanly
- Dot file marker for local modules: Deferred in ADR; current impl uses `@` for registry, `.`/`/` for local
---
## 3. Config Parsing for modules{} and registry{} Blocks
**Research Question**: How to add new config DSL blocks?
**Decision**: Create ModulesConfig and RegistryConfig classes implementing ConfigScope interface
**Rationale**:
- ConfigScope is an ExtensionPoint (pf4j) that ConfigBuilder automatically discovers
- Classes implementing ConfigScope and annotated with @ScopeName are automatically parsed
- No need to modify ConfigBuilder or create custom DSL parsers
- Pattern used throughout Nextflow: FusionConfig, CondaConfig, DockerConfig, etc.
- Provides type safety via @CompileStatic and validation via @ConfigOption
**Reference Implementation**:
```
Location: modules/nextflow/src/main/groovy/nextflow/fusion/FusionConfig.groovy
Pattern:
@ScopeName("modules")
@Description("Module version declarations")
@CompileStatic
class ModulesConfig implements ConfigScope {
@ConfigOption
@Description("Module version mappings")
final Map<String, String> modules = [:]
ModulesConfig() {}
ModulesConfig(Map opts) {
// Parse from config map
}
}
```
**ConfigScope Interface**:
```
Location: modules/nf-lang/src/main/java/nextflow/config/spec/ConfigScope.java
public interface ConfigScope extends ExtensionPoint {}
```
**RegistryConfig Pattern**:
```groovy
@ScopeName("registry")
@Description("Module registry configuration")
@CompileStatic
class RegistryConfig implements ConfigScope {
static final String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api'
@ConfigOption
final Collection<String> url // One or more URLs in priority order
@ConfigOption
final String apiKey // API key; falls back to NXF_REGISTRY_TOKEN env var
RegistryConfig() {
url = [DEFAULT_REGISTRY_URL]
apiKey = null
}
RegistryConfig(Map opts) {
url = opts.url ?: [DEFAULT_REGISTRY_URL]
apiKey = opts.apiKey as String
}
String getUrl() { url ? url[0] : DEFAULT_REGISTRY_URL }
Collection<String> getAllUrls() { url ?: [DEFAULT_REGISTRY_URL] }
String getApiKey() { apiKey ?: SysEnv.get('NXF_REGISTRY_TOKEN') }
}
```
**Integration Point**: ConfigBuilder automatically discovers and parses ConfigScope implementations via ExtensionPoint mechanism
**Alternatives Considered**:
- Custom DSL parsers (ModulesDsl/RegistryDsl): Rejected - unnecessary complexity, ConfigScope pattern handles this automatically
- JSON/YAML config file: Rejected - inconsistent with Nextflow config style
- Dedicated pipeline.yaml: Deferred per ADR Open Questions
---
## 4. Registry HTTP Communication
**Research Question**: How to communicate with module registry API?
**Decision**: Create HttpModuleRepository following HttpPluginRepository pattern
**Rationale**:
- HttpPluginRepository provides robust HTTP client with retry logic
- Uses HxClient from io.seqera.http (already a dependency)
- Handles authentication headers consistently
- Supports connection pooling and timeout configuration
**Reference Implementation**:
```
Location: modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy
Pattern:
class HttpModuleRepository {
private final URI url
private final HxClient httpClient
private final String authToken
ModuleInfo getModule(String name, String version)
List<ModuleInfo> search(String query, int limit)
Path download(String name, String version, Path target)
void publish(String name, Path bundle)
}
```
**API Endpoints** (from ADR):
```
GET /api/modules?query=<text> # Search
GET /api/modules/{name} # Get module + latest release
GET /api/modules/{name}/releases # List all releases
GET /api/modules/{name}/{version} # Get specific release
GET /api/modules/{name}/{version}/download # Download bundle
POST /api/modules/{name} # Publish (authenticated)
```
**Alternatives Considered**:
- Direct HttpClient usage: Rejected - loses retry, pooling benefits
- gRPC protocol: Rejected - registry already uses REST
---
## 5. Authentication Patterns
**Research Question**: How to handle registry authentication?
**Decision**: Support `NXF_REGISTRY_TOKEN` env var + `registry.apiKey` config field
**Rationale**:
- Environment variable provides CI/CD compatibility
- `apiKey` config field allows explicit token configuration
- Authentication is only applied to the primary (first) registry URL
- Bearer token in Authorization header (standard HTTP auth)
**Implementation**:
```
RegistryConfig.getApiKey() returns:
1. registry.apiKey config value if set
2. NXF_REGISTRY_TOKEN environment variable as fallback
3. null if neither is set (unauthenticated requests)
```
**Config Syntax**:
```nextflow
registry {
apiKey = '${NXF_REGISTRY_TOKEN}'
}
```
**Alternatives Considered**:
- Per-registry token map (`auth {}` block): Was in initial design; simplified to single `apiKey` since only the primary registry uses authentication
- Secrets file (~/.nextflow/secrets.json): Possible future enhancement
- OAuth flow: Rejected for CLI — token-based simpler
---
## 6. Checksum Verification
**Research Question**: How to implement module integrity verification?
**Decision**: SHA-256 checksum stored in `.checksum` file, verified on every run
**Rationale**:
- SHA-256 is industry standard, already used for plugin verification
- `.checksum` file stores registry-provided checksum (from X-Checksum header)
- Local checksum computed on-demand and compared
- Mismatch indicates local modification (warn, don't override)
**Implementation Pattern**:
```groovy
class ModuleChecksum {
static final String ALGORITHM = 'SHA-256'
static String compute(Path moduleDir) {
// Hash all files in module directory
// Exclude .checksum itself
// Return hex-encoded SHA-256
}
static boolean verify(Path moduleDir) {
def expected = moduleDir.resolve('.checksum').text.trim()
def actual = compute(moduleDir)
return expected == actual
}
static void save(Path moduleDir, String checksum) {
moduleDir.resolve('.checksum').text = checksum
}
}
```
**Checksum Scope**: Covers all files in module directory (main.nf, meta.yaml, README.md, etc.)
**Alternatives Considered**:
- Per-file checksums: Rejected - adds complexity, single checksum sufficient
- MD5: Rejected - SHA-256 more secure
---
## 7. Version Constraint Syntax
**Research Question**: What version constraint syntax to use for module dependencies?
**Decision**: Reuse existing Nextflow plugin version constraint syntax
**Rationale**:
- Already implemented and tested in plugin system
- Users familiar with existing `nextflowVersion` syntax
- Supports ranges, comparisons, exact versions
- No new parser code needed
**Supported Syntax**:
| Notation | Meaning | Example |
|----------|---------|---------|
| `1.2.3` | Exact version | `@nf-core/fastqc@1.0.0` |
| `>=1.2.3` | Greater or equal | `@nf-core/fastqc@>=1.0.0` |
| `<=1.2.3` | Less or equal | `@nf-core/fastqc@<=2.0.0` |
| `>=1.2.0,<2.0.0` | Range | `@nf-core/samtools@>=1.0.0,<2.0.0` |
**Reference**: Version parsing code exists in plugin system; reuse VersionNumber class
**Alternatives Considered**:
- NPM-style `^` and `~`: Rejected - inconsistent with existing Nextflow patterns
- Always latest: Rejected - breaks reproducibility
---
## 8. Tool Arguments Implementation
> **⚠️ REMOVED FROM ADR** — The tool arguments feature (`tools.<name>.args` in meta.yaml and process config) was removed from the module system ADR. It is not implemented and not planned in the current scope. The `meta.yaml` format used in the actual implementation (`ModuleSpec`) does not include tool/argument definitions.
---
## Summary of Key Decisions
| Area | Decision | Key Reference |
|------|----------|---------------|
| CLI | JCommander subcommands; each extends CmdBase (ModuleRun extends CmdRun) | CmdModule.groovy |
| DSL Parser | SPI pattern — ResolveIncludeVisitor delegates to RemoteModuleResolver; DefaultRemoteModuleResolver bridges to ModuleResolver | ResolveIncludeVisitor.java, RemoteModuleResolver.java |
| Config | ModulesConfig + RegistryConfig (ConfigScope) | FusionConfig.groovy, ConfigScope.java |
| Registry HTTP | ModuleRegistryClient using HxClient + npr-api models | HttpPluginRepository.groovy |
| Authentication | `NXF_REGISTRY_TOKEN` env var or `registry.apiKey` config field (primary registry only) | RegistryConfig.groovy |
| Checksums | SHA-256/SHA-512, `.checksum` file, download integrity via X-Checksum header | ModuleChecksum.groovy |
| Version Storage | `nextflow_spec.json` (auto-managed); `modules {}` in nextflow.config (manual alternative) | PipelineSpec.groovy |
| Version Syntax | Plugin-compatible constraints | VersionNumber class |
| Tool Args | ~~Implicit variable, parse-time validation~~ — **Removed from ADR** | N/A |
---
## Open Items (Deferred)
1. **Local vs managed module distinction**: Resolved — `@` prefix for registry modules only; local paths start with `.` or `/`
2. **Tool arguments**: Removed from ADR — not in scope
3. **Module version location**: Resolved — `nextflow_spec.json` (auto-managed by `module install`); `modules {}` block in `nextflow.config` supported as alternative
4. **DSL parser `@scope/name` include**: ✅ Resolved — SPI pattern implemented (T017a-d)

261
nextflow/specs/251117-module-system/spec.md Normal file
View File

@@ -0,0 +1,261 @@
# Feature Specification: Nextflow Module System Client
**Feature Branch**: `251117-module-system`
**Created**: 2026-01-15
**Status**: Draft
**Input**: User description: "Implement Nextflow module system client based on ADR 20251114-module-system.md. Focus on client-side implementation only - CLI commands, DSL parser extensions, dependency resolution, and local storage. Registry backend is assumed to be already implemented."
## Overview
This specification covers the **Nextflow client-side implementation** of the module system, enabling pipeline developers to:
- Include remote modules from the Nextflow registry using `@scope/name` syntax
- Manage module versions through `nextflow.config`
- Use CLI commands to install, search, list, remove, publish, and run modules
- Configure module parameters through structured `meta.yaml` definitions
**Out of Scope**: Registry backend implementation (assumed already available at `registry.nextflow.io`)
## User Scenarios & Testing
### User Story 1 - Install and Use Registry Module (Priority: P1)
A pipeline developer wants to use a pre-built module from the Nextflow registry in their workflow without manually downloading or managing module files.
**Why this priority**: This is the core value proposition - enabling code reuse from the ecosystem. Without this, the module system provides no benefit.
**Independent Test**: Can be fully tested by running `nextflow module install nf-core/fastqc` and then executing a workflow that includes the module. Delivers immediate value by enabling module consumption.
**Acceptance Scenarios**:
1. **Given** a new Nextflow project with no modules installed, **When** user runs `nextflow module install nf-core/fastqc`, **Then** the module is downloaded to `modules/@nf-core/fastqc/`, a `.checksum` file is created, and `nextflow_spec.json` is updated with the version
2. **Given** a workflow file with `include { FASTQC } from '@nf-core/fastqc'`, **When** user runs `nextflow run main.nf`, **Then** Nextflow resolves the module from local storage and executes the process
3. **Given** a module version declared in `nextflow.config`, **When** user includes the module, **Then** the declared version is used (not latest)
---
### User Story 2 - Run Module Directly (Priority: P1)
A user wants to run a module directly from the command line without writing a wrapper workflow.
**Why this priority**: Enables immediate productivity - users can test and execute modules without boilerplate code, essential for AI agents and quick experimentation.
**Independent Test**: Can be tested by running `nextflow module run nf-core/fastqc --input 'data/*.fq'` and verifying the process executes.
**Acceptance Scenarios**:
1. **Given** a module is available (locally or in registry), **When** user runs `nextflow module run nf-core/fastqc --input 'data/*.fastq'`, **Then** the module is executed with the provided inputs mapped to process parameters
2. **Given** a module with parameters defined in `meta.yaml`, **When** user runs `nextflow module run nf-core/bwa-align --batch_size 100000`, **Then** the parameter is validated and passed to the process
3. **Given** a module is not installed locally, **When** user runs `nextflow module run nf-core/salmon`, **Then** the module is automatically downloaded before execution
---
### User Story 3 - Module Parameters (Priority: P1)
A module author wants to define typed, documented parameters that provide a clear interface for module customization.
**Why this priority**: Critical for module usability - provides type-safe, documented parameters that enable IDE autocompletion and validation, replacing the opaque `ext.args` pattern.
**Independent Test**: Can be tested by configuring `params.batch_size = 100000` in config and verifying the parameter is applied in the script.
**Acceptance Scenarios**:
1. **Given** a module with `params` defined in `meta.yaml`, **When** user configures `params.batch_size = 100000` in config, **Then** the parameter is accessible in scripts via `params.batch_size`
2. **Given** a parameter with type validation, **When** user provides an invalid value type, **Then** a validation error is displayed
3. **Given** a module with documented parameters, **When** user runs `nextflow module run --help`, **Then** available parameters with descriptions are listed
---
### User Story 4 - Module Version Management (Priority: P2)
A pipeline developer wants to pin and manage module versions to ensure reproducible workflow executions.
**Why this priority**: Reproducibility is important for scientific workflows - version pinning ensures consistent results.
**Independent Test**: Can be tested by modifying `nextflow.config` module versions and verifying the correct version is used on workflow run.
**Acceptance Scenarios**:
1. **Given** a module is installed at version 1.0.0, **When** user changes `nextflow_spec.json` to specify version 1.1.0 and runs the workflow, **Then** version 1.1.0 is automatically downloaded and replaces the local copy
2. **Given** modules installed locally, **When** user runs `nextflow module list`, **Then** configured version, installed version, latest available version, and status are displayed for each module
---
### User Story 5 - Module Integrity Protection (Priority: P2)
A pipeline developer who has locally modified a module (for debugging or customization) wants to be protected from accidentally losing those changes.
**Why this priority**: Protects user work - important for developer experience but not blocking core functionality.
**Independent Test**: Can be tested by modifying a module's `main.nf` locally, then attempting to install a different version and verifying the warning appears.
**Acceptance Scenarios**:
1. **Given** a locally modified module (checksum mismatch with `.checksum`), **When** user tries to install a different version, **Then** Nextflow warns about local modifications and does NOT override
2. **Given** a locally modified module, **When** user runs `nextflow module install -force`, **Then** the local module is replaced with the registry version
3. **Given** a locally modified module, **When** user runs the workflow, **Then** a warning is displayed about checksum mismatch but execution continues
---
### User Story 6 - Remove Module (Priority: P3)
A pipeline developer wants to remove a module they no longer need.
**Why this priority**: Housekeeping feature - useful but not blocking core workflows.
**Independent Test**: Can be tested by running `nextflow module remove nf-core/fastqc` and verifying files are deleted and config is updated.
**Acceptance Scenarios**:
1. **Given** a module is installed, **When** user runs `nextflow module remove nf-core/fastqc`, **Then** the module directory is deleted and the entry is removed from `nextflow_spec.json`
2. **Given** a module is referenced in workflow files, **When** user runs `nextflow module remove`, **Then** a warning is displayed about the reference but removal proceeds
---
### User Story 7 - Search and Discover Modules (Priority: P3)
A pipeline developer wants to find available modules in the registry that match their analysis needs.
**Why this priority**: Discovery feature - useful but users can find modules through documentation or registry web UI.
**Independent Test**: Can be tested by running `nextflow module search bwa` and verifying results are displayed with name, version, and description.
**Acceptance Scenarios**:
1. **Given** modules exist in the registry, **When** user runs `nextflow module search alignment`, **Then** matching modules are displayed with name, latest version, description, and download count
2. **Given** user wants JSON output for scripting, **When** user runs `nextflow module search fastqc -json`, **Then** results are returned in parseable JSON format
3. **Given** many results exist, **When** user runs `nextflow module search quality -limit 5`, **Then** only 5 results are returned
---
### User Story 8 - Publish Module to Registry (Priority: P3)
A module author wants to publish their module to the Nextflow registry for others to use.
**Why this priority**: Ecosystem contribution feature - important for growth but users can consume modules without publishing capability.
**Independent Test**: Can be tested by creating a valid module structure and running `nextflow module publish -dry-run` to validate.
**Acceptance Scenarios**:
1. **Given** a valid module with `main.nf`, `meta.yaml`, and `README.md`, **When** user runs `nextflow module publish myorg/my-module`, **Then** the module is uploaded to the registry and becomes available for installation
2. **Given** an invalid module (missing required fields), **When** user runs `nextflow module publish`, **Then** validation errors are displayed listing the missing requirements
3. **Given** no authentication configured, **When** user runs `nextflow module publish`, **Then** a clear error message indicates authentication is required
---
### Edge Cases
- What happens when the registry is unreachable during module resolution?
- Nextflow uses locally cached modules if available, otherwise fails with a clear network error
- How does the system handle circular module dependencies?
- Dependency resolver detects cycles and fails with an error listing the cycle
- What happens when two modules require incompatible versions of the same dependency?
- System automatically selects the highest compatible version; if no compatible version exists, fails with error listing conflicting requirements
- How are modules resolved when multiple registries are configured?
- Registries are tried in order; first match wins
- What happens when `meta.yaml` is missing from a module?
- Module is treated as having no dependencies; basic functionality works
- What happens when local module directory is corrupted or incomplete?
- Checksum mismatch triggers warning; `-force` allows re-download
## Requirements
### Functional Requirements
#### DSL Parser Extension
- **FR-001**: System MUST recognize `@scope/name` syntax in `include` statements as registry module references
- **FR-002**: System MUST distinguish between local file paths (starting with `.` or `/`) and registry modules (starting with `@`)
- **FR-003**: System MUST resolve module versions from `nextflow_spec.json` before downloading
- **FR-004**: System MUST parse and validate `meta.yaml` files for module metadata and dependencies
#### Module Resolution
- **FR-005**: System MUST resolve modules at workflow parse time (after plugin resolution)
- **FR-006**: System MUST check local `modules/@scope/name/` directory before querying registry
- **FR-007**: System MUST verify module integrity using `.checksum` file on every run
- **FR-008**: System MUST download modules from registry when not present locally or when version differs
- **FR-009**: System MUST NOT override locally modified modules (checksum mismatch) unless `-force` is used
- **FR-010**: System MUST resolve version conflicts by selecting the highest compatible version; if no compatible version exists, MUST fail with error listing conflicting requirements
#### Local Storage
- **FR-011**: System MUST store modules in `modules/@scope/name/` directory structure (single version per module)
- **FR-012**: System MUST create `.checksum` file from registry's X-Checksum header on download
- **FR-013**: System MUST store module's `main.nf`, `meta.yaml`, and supporting files in the module directory
#### CLI Commands
- **FR-014**: System MUST provide `nextflow module install [scope/name]` command to download modules
- **FR-015**: System MUST provide `nextflow module search <query>` command to search the registry
- **FR-016**: System MUST provide `nextflow module list` command to show installed vs configured modules
- **FR-017**: System MUST provide `nextflow module remove scope/name` command to delete modules
- **FR-018**: System MUST provide `nextflow module publish scope/name` command to upload modules to registry
- **FR-019**: System MUST provide `nextflow module run scope/name` command to execute modules directly
- **FR-019b**: System MUST provide `nextflow module info scope/name` command to display module metadata and a usage template
#### Configuration
- **FR-020**: System MUST persist module versions in `nextflow_spec.json`; MUST also read versions from `modules {}` block in `nextflow.config` as an alternative
- **FR-021**: System MUST support `registry {}` block with `url` and `apiKey` fields for configuring registry URL and authentication
- **FR-022**: System MUST support `NXF_REGISTRY_TOKEN` environment variable as fallback for `registry.apiKey`
- **FR-023**: System MUST support multiple registry URLs with fallback ordering
#### Module Parameters
- **FR-024**: System MUST parse module parameters from `params` section in `meta.yaml`
- **FR-025**: System MUST validate module parameters against `meta.yaml` schema (type) at workflow parse time
- **FR-026**: System MUST support boolean, integer, float, string, file, and path parameter types
- **FR-027**: System MUST make module parameters accessible via standard `params` variable in scripts
#### Registry Communication
- **FR-028**: System MUST communicate with registry via documented Module API endpoints
- **FR-029**: System MUST handle authentication using Bearer token in Authorization header
- **FR-030**: System MUST verify SHA-256 checksum on module download
### Key Entities
- **Module**: A reusable Nextflow process definition with `main.nf` entry point, optional `meta.yaml` manifest, and README documentation
- **Module Reference**: A scoped identifier (`@scope/name`) pointing to a registry module
- **Module Manifest (meta.yaml)**: YAML file containing module metadata, version, dependencies, and parameter definitions
- **Module Parameter**: A configurable parameter defined in `meta.yaml` with name, optional type, description, and example
- **Checksum File (.checksum)**: Local cache of registry checksum for integrity verification
- **Registry Configuration**: Settings for registry URL, authentication, and fallback ordering
## Success Criteria
### Measurable Outcomes
- **SC-001**: Pipeline developers can install and use a registry module within 5 minutes of starting a new project
- **SC-002**: Module resolution adds less than 2 seconds to workflow startup time when modules are cached locally
- **SC-003**: Users can successfully search, install, and run any module from the registry without reading documentation
- **SC-004**: 100% of module version changes in `nextflow.config` result in automatic module updates without manual intervention
- **SC-005**: Users receive clear, actionable error messages for all failure scenarios (network, validation, authentication)
- **SC-006**: Module authors can publish a new module version within 3 minutes using the CLI
- **SC-007**: Locally modified modules are never accidentally overwritten during normal operations
## Assumptions
- Registry backend is fully implemented and available at `registry.nextflow.io` with the Module API as documented in the ADR
- Existing plugin authentication system can be reused for module registry authentication
- Module bundle size limit of 1MB (uncompressed) is enforced by the registry
- Network connectivity is available for initial module downloads; offline operation uses local cache only
- The `modules/` directory is intended to be committed to the pipeline's git repository
- Version constraints in `meta.yaml` follow the same syntax as existing Nextflow plugin version constraints
- SHA-256 is used for all checksum operations
- Module parameters use standard `--<param_name>` CLI syntax
## Dependencies
- Registry backend API (Module API endpoints as specified in ADR)
- Existing Nextflow plugin system (for authentication reuse)
- Existing DSL parser infrastructure (for `include` statement extension)
- Existing config parser (for `modules {}` and `registry {}` blocks)
## Clarifications
### Session 2026-01-19
- Q: What should happen when incompatible dependency versions are detected? → A: Use highest compatible version automatically, warn if none exists
- Q: When should module parameter validation occur? → A: At workflow parse time (early, before any execution)