ma/nextflow/specs/251117-module-system/research.md

# Research: Nextflow Module System Client

**Date**: 2026-01-19
**Feature**: 251117-module-system

## Overview

This document captures technical research and decisions for implementing the Nextflow module system client. All NEEDS CLARIFICATION items from Technical Context have been resolved through codebase exploration.

---

## 1. CLI Command Structure

**Research Question**: How should `nextflow module` CLI commands be implemented?

**Decision**: JCommander native subcommands — each subcommand extends `CmdBase` directly; no trait needed

**Rationale**:
- JCommander's subcommand support handles parameter parsing automatically per subcommand
- Each subcommand (install, run, list, remove, search, info, publish) is a separate class extending CmdBase
- `ModuleRun` extends `CmdRun` to reuse pipeline execution logic (PR #6381)
- No custom `ModuleSubCmd` trait needed; cleaner architecture
- `CmdModule` is registered in `Launcher` alongside all other top-level commands

**Implemented Pattern**:
```groovy
@Parameters(commandDescription = "Manage Nextflow modules")
class CmdModule extends CmdBase implements UsageAware {
    static final List<CmdBase> commands = []

    static {
        commands << new ModuleInstall()   // extends CmdBase
        commands << new ModuleRun()       // extends CmdRun
        commands << new ModuleList()      // extends CmdBase
        commands << new ModuleRemove()    // extends CmdBase
        commands << new ModuleSearch()    // extends CmdBase
        commands << new ModuleInfo()      // extends CmdBase
        commands << new ModulePublish()   // extends CmdBase
    }

    void run() {
        final jc = commander()    // JCommander with all subcommands registered
        jc.parse(args as String[])
        final subcommand = jc.getCommands().get(jc.getParsedCommand()).getObjects()[0]
        subcommand.run()
    }
}
```

**Alternatives Considered**:
- CmdFs trait pattern: Considered initially; replaced by JCommander native subcommands — simpler and avoids custom parsing
- Separate top-level Cmd classes (CmdModuleInstall, etc.): Rejected — too many entry points
- Plugin-based CLI extension: Rejected — module system is core functionality, not optional

---

## 2. DSL Parser Extension for @scope/name

**Research Question**: How to extend `include` statement parsing for registry modules?

**Decision**: Extend `ResolveIncludeVisitor` to detect `@` prefix and delegate to a `RemoteModuleResolver` SPI loaded via Java `ServiceLoader`

**Rationale**:
- Keeps `nf-lang` decoupled from runtime module resolution (`nf-lang` has no dependency on `nextflow` module)
- SPI pattern allows plugins or custom implementations to override the default resolver
- Detection: `source.startsWith('@')` distinguishes registry vs local paths — preserves existing include behavior
- Resolution at parse time (after plugin resolution) per ADR

**Implemented Architecture**:
```
include { X } from '@scope/name'
      ↓
ResolveIncludeVisitor.visitInclude()  [nf-lang]
  source.startsWith("@") → RemoteModuleResolverProvider.getInstance().resolve(source, baseDir)
      ↓
RemoteModuleResolverProvider  [nf-lang]
  Java ServiceLoader discovers implementations; picks highest priority
      ↓
DefaultRemoteModuleResolver  [nextflow module]
  Calls ModuleResolver.installModule(reference, version, autoInstall=true)
  Returns Path to modules/@scope/name/main.nf
```

**Key Files**:
- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolver.java` — SPI interface
- `modules/nf-lang/src/main/java/nextflow/module/spi/RemoteModuleResolverProvider.java` — ServiceLoader singleton
- `modules/nf-lang/src/main/java/nextflow/module/spi/FallbackRemoteModuleResolver.java` — error fallback
- `modules/nf-lang/src/main/java/nextflow/script/control/ResolveIncludeVisitor.java` — MODIFIED
- `modules/nextflow/src/main/groovy/nextflow/module/DefaultRemoteModuleResolver.groovy` — default impl

**Alternatives Considered**:
- New ANTLR grammar token for `@`: Rejected — unnecessary parser complexity
- Direct dependency from nf-lang to nextflow module: Rejected — circular dependency risk; SPI decouples cleanly
- Dot file marker for local modules: Deferred in ADR; current impl uses `@` for registry, `.`/`/` for local

---

## 3. Config Parsing for modules{} and registry{} Blocks

**Research Question**: How to add new config DSL blocks?

**Decision**: Create ModulesConfig and RegistryConfig classes implementing ConfigScope interface

**Rationale**:
- ConfigScope is an ExtensionPoint (pf4j) that ConfigBuilder automatically discovers
- Classes implementing ConfigScope and annotated with @ScopeName are automatically parsed
- No need to modify ConfigBuilder or create custom DSL parsers
- Pattern used throughout Nextflow: FusionConfig, CondaConfig, DockerConfig, etc.
- Provides type safety via @CompileStatic and validation via @ConfigOption

**Reference Implementation**:
```
Location: modules/nextflow/src/main/groovy/nextflow/fusion/FusionConfig.groovy
Pattern:
  @ScopeName("modules")
  @Description("Module version declarations")
  @CompileStatic
  class ModulesConfig implements ConfigScope {
      @ConfigOption
      @Description("Module version mappings")
      final Map<String, String> modules = [:]

      ModulesConfig() {}

      ModulesConfig(Map opts) {
          // Parse from config map
      }
  }
```

**ConfigScope Interface**:
```
Location: modules/nf-lang/src/main/java/nextflow/config/spec/ConfigScope.java
public interface ConfigScope extends ExtensionPoint {}
```

**RegistryConfig Pattern**:
```groovy
@ScopeName("registry")
@Description("Module registry configuration")
@CompileStatic
class RegistryConfig implements ConfigScope {
    static final String DEFAULT_REGISTRY_URL = 'https://registry.nextflow.io/api'

    @ConfigOption
    final Collection<String> url   // One or more URLs in priority order

    @ConfigOption
    final String apiKey            // API key; falls back to NXF_REGISTRY_TOKEN env var

    RegistryConfig() {
        url = [DEFAULT_REGISTRY_URL]
        apiKey = null
    }

    RegistryConfig(Map opts) {
        url = opts.url ?: [DEFAULT_REGISTRY_URL]
        apiKey = opts.apiKey as String
    }

    String getUrl() { url ? url[0] : DEFAULT_REGISTRY_URL }
    Collection<String> getAllUrls() { url ?: [DEFAULT_REGISTRY_URL] }
    String getApiKey() { apiKey ?: SysEnv.get('NXF_REGISTRY_TOKEN') }
}
```

**Integration Point**: ConfigBuilder automatically discovers and parses ConfigScope implementations via ExtensionPoint mechanism

**Alternatives Considered**:
- Custom DSL parsers (ModulesDsl/RegistryDsl): Rejected - unnecessary complexity, ConfigScope pattern handles this automatically
- JSON/YAML config file: Rejected - inconsistent with Nextflow config style
- Dedicated pipeline.yaml: Deferred per ADR Open Questions

---

## 4. Registry HTTP Communication

**Research Question**: How to communicate with module registry API?

**Decision**: Create HttpModuleRepository following HttpPluginRepository pattern

**Rationale**:
- HttpPluginRepository provides robust HTTP client with retry logic
- Uses HxClient from io.seqera.http (already a dependency)
- Handles authentication headers consistently
- Supports connection pooling and timeout configuration

**Reference Implementation**:
```
Location: modules/nf-commons/src/main/nextflow/plugin/HttpPluginRepository.groovy
Pattern:
  class HttpModuleRepository {
      private final URI url
      private final HxClient httpClient
      private final String authToken

      ModuleInfo getModule(String name, String version)
      List<ModuleInfo> search(String query, int limit)
      Path download(String name, String version, Path target)
      void publish(String name, Path bundle)
  }
```

**API Endpoints** (from ADR):
```
GET  /api/modules?query=<text>               # Search
GET  /api/modules/{name}                     # Get module + latest release
GET  /api/modules/{name}/releases            # List all releases
GET  /api/modules/{name}/{version}           # Get specific release
GET  /api/modules/{name}/{version}/download  # Download bundle
POST /api/modules/{name}                     # Publish (authenticated)
```

**Alternatives Considered**:
- Direct HttpClient usage: Rejected - loses retry, pooling benefits
- gRPC protocol: Rejected - registry already uses REST

---

## 5. Authentication Patterns

**Research Question**: How to handle registry authentication?

**Decision**: Support `NXF_REGISTRY_TOKEN` env var + `registry.apiKey` config field

**Rationale**:
- Environment variable provides CI/CD compatibility
- `apiKey` config field allows explicit token configuration
- Authentication is only applied to the primary (first) registry URL
- Bearer token in Authorization header (standard HTTP auth)

**Implementation**:
```
RegistryConfig.getApiKey() returns:
  1. registry.apiKey config value if set
  2. NXF_REGISTRY_TOKEN environment variable as fallback
  3. null if neither is set (unauthenticated requests)
```

**Config Syntax**:
```nextflow
registry {
    apiKey = '${NXF_REGISTRY_TOKEN}'
}
```

**Alternatives Considered**:
- Per-registry token map (`auth {}` block): Was in initial design; simplified to single `apiKey` since only the primary registry uses authentication
- Secrets file (~/.nextflow/secrets.json): Possible future enhancement
- OAuth flow: Rejected for CLI — token-based simpler

---

## 6. Checksum Verification

**Research Question**: How to implement module integrity verification?

**Decision**: SHA-256 checksum stored in `.checksum` file, verified on every run

**Rationale**:
- SHA-256 is industry standard, already used for plugin verification
- `.checksum` file stores registry-provided checksum (from X-Checksum header)
- Local checksum computed on-demand and compared
- Mismatch indicates local modification (warn, don't override)

**Implementation Pattern**:
```groovy
class ModuleChecksum {
    static final String ALGORITHM = 'SHA-256'

    static String compute(Path moduleDir) {
        // Hash all files in module directory
        // Exclude .checksum itself
        // Return hex-encoded SHA-256
    }

    static boolean verify(Path moduleDir) {
        def expected = moduleDir.resolve('.checksum').text.trim()
        def actual = compute(moduleDir)
        return expected == actual
    }

    static void save(Path moduleDir, String checksum) {
        moduleDir.resolve('.checksum').text = checksum
    }
}
```

**Checksum Scope**: Covers all files in module directory (main.nf, meta.yaml, README.md, etc.)

**Alternatives Considered**:
- Per-file checksums: Rejected - adds complexity, single checksum sufficient
- MD5: Rejected - SHA-256 more secure

---

## 7. Version Constraint Syntax

**Research Question**: What version constraint syntax to use for module dependencies?

**Decision**: Reuse existing Nextflow plugin version constraint syntax

**Rationale**:
- Already implemented and tested in plugin system
- Users familiar with existing `nextflowVersion` syntax
- Supports ranges, comparisons, exact versions
- No new parser code needed

**Supported Syntax**:
| Notation | Meaning | Example |
|----------|---------|---------|
| `1.2.3` | Exact version | `@nf-core/fastqc@1.0.0` |
| `>=1.2.3` | Greater or equal | `@nf-core/fastqc@>=1.0.0` |
| `<=1.2.3` | Less or equal | `@nf-core/fastqc@<=2.0.0` |
| `>=1.2.0,<2.0.0` | Range | `@nf-core/samtools@>=1.0.0,<2.0.0` |

**Reference**: Version parsing code exists in plugin system; reuse VersionNumber class

**Alternatives Considered**:
- NPM-style `^` and `~`: Rejected - inconsistent with existing Nextflow patterns
- Always latest: Rejected - breaks reproducibility

---

## 8. Tool Arguments Implementation

> **⚠️ REMOVED FROM ADR** — The tool arguments feature (`tools.<name>.args` in meta.yaml and process config) was removed from the module system ADR. It is not implemented and not planned in the current scope. The `meta.yaml` format used in the actual implementation (`ModuleSpec`) does not include tool/argument definitions.

---

## Summary of Key Decisions

| Area | Decision | Key Reference |
|------|----------|---------------|
| CLI | JCommander subcommands; each extends CmdBase (ModuleRun extends CmdRun) | CmdModule.groovy |
| DSL Parser | SPI pattern — ResolveIncludeVisitor delegates to RemoteModuleResolver; DefaultRemoteModuleResolver bridges to ModuleResolver | ResolveIncludeVisitor.java, RemoteModuleResolver.java |
| Config | ModulesConfig + RegistryConfig (ConfigScope) | FusionConfig.groovy, ConfigScope.java |
| Registry HTTP | ModuleRegistryClient using HxClient + npr-api models | HttpPluginRepository.groovy |
| Authentication | `NXF_REGISTRY_TOKEN` env var or `registry.apiKey` config field (primary registry only) | RegistryConfig.groovy |
| Checksums | SHA-256/SHA-512, `.checksum` file, download integrity via X-Checksum header | ModuleChecksum.groovy |
| Version Storage | `nextflow_spec.json` (auto-managed); `modules {}` in nextflow.config (manual alternative) | PipelineSpec.groovy |
| Version Syntax | Plugin-compatible constraints | VersionNumber class |
| Tool Args | ~~Implicit variable, parse-time validation~~ — **Removed from ADR** | N/A |

---

## Open Items (Deferred)

1. **Local vs managed module distinction**: Resolved — `@` prefix for registry modules only; local paths start with `.` or `/`
2. **Tool arguments**: Removed from ADR — not in scope
3. **Module version location**: Resolved — `nextflow_spec.json` (auto-managed by `module install`); `modules {}` block in `nextflow.config` supported as alternative
4. **DSL parser `@scope/name` include**: ✅ Resolved — SPI pattern implemented (T017a-d)