add nextflow d30e48d
This commit is contained in:
184
nextflow/.claude/commands/speckit.analyze.md
Normal file
184
nextflow/.claude/commands/speckit.analyze.md
Normal file
@@ -0,0 +1,184 @@
|
||||
---
|
||||
description: Perform a non-destructive cross-artifact consistency and quality analysis across spec.md, plan.md, and tasks.md after task generation.
|
||||
---
|
||||
|
||||
## User Input
|
||||
|
||||
```text
|
||||
$ARGUMENTS
|
||||
```
|
||||
|
||||
You **MUST** consider the user input before proceeding (if not empty).
|
||||
|
||||
## Goal
|
||||
|
||||
Identify inconsistencies, duplications, ambiguities, and underspecified items across the three core artifacts (`spec.md`, `plan.md`, `tasks.md`) before implementation. This command MUST run only after `/speckit.tasks` has successfully produced a complete `tasks.md`.
|
||||
|
||||
## Operating Constraints
|
||||
|
||||
**STRICTLY READ-ONLY**: Do **not** modify any files. Output a structured analysis report. Offer an optional remediation plan (user must explicitly approve before any follow-up editing commands would be invoked manually).
|
||||
|
||||
**Constitution Authority**: The project constitution (`.specify/memory/constitution.md`) is **non-negotiable** within this analysis scope. Constitution conflicts are automatically CRITICAL and require adjustment of the spec, plan, or tasks—not dilution, reinterpretation, or silent ignoring of the principle. If a principle itself needs to change, that must occur in a separate, explicit constitution update outside `/speckit.analyze`.
|
||||
|
||||
## Execution Steps
|
||||
|
||||
### 1. Initialize Analysis Context
|
||||
|
||||
Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks` once from repo root and parse JSON for FEATURE_DIR and AVAILABLE_DOCS. Derive absolute paths:
|
||||
|
||||
- SPEC = FEATURE_DIR/spec.md
|
||||
- PLAN = FEATURE_DIR/plan.md
|
||||
- TASKS = FEATURE_DIR/tasks.md
|
||||
|
||||
Abort with an error message if any required file is missing (instruct the user to run missing prerequisite command).
|
||||
For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
||||
|
||||
### 2. Load Artifacts (Progressive Disclosure)
|
||||
|
||||
Load only the minimal necessary context from each artifact:
|
||||
|
||||
**From spec.md:**
|
||||
|
||||
- Overview/Context
|
||||
- Functional Requirements
|
||||
- Non-Functional Requirements
|
||||
- User Stories
|
||||
- Edge Cases (if present)
|
||||
|
||||
**From plan.md:**
|
||||
|
||||
- Architecture/stack choices
|
||||
- Data Model references
|
||||
- Phases
|
||||
- Technical constraints
|
||||
|
||||
**From tasks.md:**
|
||||
|
||||
- Task IDs
|
||||
- Descriptions
|
||||
- Phase grouping
|
||||
- Parallel markers [P]
|
||||
- Referenced file paths
|
||||
|
||||
**From constitution:**
|
||||
|
||||
- Load `.specify/memory/constitution.md` for principle validation
|
||||
|
||||
### 3. Build Semantic Models
|
||||
|
||||
Create internal representations (do not include raw artifacts in output):
|
||||
|
||||
- **Requirements inventory**: Each functional + non-functional requirement with a stable key (derive slug based on imperative phrase; e.g., "User can upload file" → `user-can-upload-file`)
|
||||
- **User story/action inventory**: Discrete user actions with acceptance criteria
|
||||
- **Task coverage mapping**: Map each task to one or more requirements or stories (inference by keyword / explicit reference patterns like IDs or key phrases)
|
||||
- **Constitution rule set**: Extract principle names and MUST/SHOULD normative statements
|
||||
|
||||
### 4. Detection Passes (Token-Efficient Analysis)
|
||||
|
||||
Focus on high-signal findings. Limit to 50 findings total; aggregate remainder in overflow summary.
|
||||
|
||||
#### A. Duplication Detection
|
||||
|
||||
- Identify near-duplicate requirements
|
||||
- Mark lower-quality phrasing for consolidation
|
||||
|
||||
#### B. Ambiguity Detection
|
||||
|
||||
- Flag vague adjectives (fast, scalable, secure, intuitive, robust) lacking measurable criteria
|
||||
- Flag unresolved placeholders (TODO, TKTK, ???, `<placeholder>`, etc.)
|
||||
|
||||
#### C. Underspecification
|
||||
|
||||
- Requirements with verbs but missing object or measurable outcome
|
||||
- User stories missing acceptance criteria alignment
|
||||
- Tasks referencing files or components not defined in spec/plan
|
||||
|
||||
#### D. Constitution Alignment
|
||||
|
||||
- Any requirement or plan element conflicting with a MUST principle
|
||||
- Missing mandated sections or quality gates from constitution
|
||||
|
||||
#### E. Coverage Gaps
|
||||
|
||||
- Requirements with zero associated tasks
|
||||
- Tasks with no mapped requirement/story
|
||||
- Non-functional requirements not reflected in tasks (e.g., performance, security)
|
||||
|
||||
#### F. Inconsistency
|
||||
|
||||
- Terminology drift (same concept named differently across files)
|
||||
- Data entities referenced in plan but absent in spec (or vice versa)
|
||||
- Task ordering contradictions (e.g., integration tasks before foundational setup tasks without dependency note)
|
||||
- Conflicting requirements (e.g., one requires Next.js while other specifies Vue)
|
||||
|
||||
### 5. Severity Assignment
|
||||
|
||||
Use this heuristic to prioritize findings:
|
||||
|
||||
- **CRITICAL**: Violates constitution MUST, missing core spec artifact, or requirement with zero coverage that blocks baseline functionality
|
||||
- **HIGH**: Duplicate or conflicting requirement, ambiguous security/performance attribute, untestable acceptance criterion
|
||||
- **MEDIUM**: Terminology drift, missing non-functional task coverage, underspecified edge case
|
||||
- **LOW**: Style/wording improvements, minor redundancy not affecting execution order
|
||||
|
||||
### 6. Produce Compact Analysis Report
|
||||
|
||||
Output a Markdown report (no file writes) with the following structure:
|
||||
|
||||
## Specification Analysis Report
|
||||
|
||||
| ID | Category | Severity | Location(s) | Summary | Recommendation |
|
||||
|----|----------|----------|-------------|---------|----------------|
|
||||
| A1 | Duplication | HIGH | spec.md:L120-134 | Two similar requirements ... | Merge phrasing; keep clearer version |
|
||||
|
||||
(Add one row per finding; generate stable IDs prefixed by category initial.)
|
||||
|
||||
**Coverage Summary Table:**
|
||||
|
||||
| Requirement Key | Has Task? | Task IDs | Notes |
|
||||
|-----------------|-----------|----------|-------|
|
||||
|
||||
**Constitution Alignment Issues:** (if any)
|
||||
|
||||
**Unmapped Tasks:** (if any)
|
||||
|
||||
**Metrics:**
|
||||
|
||||
- Total Requirements
|
||||
- Total Tasks
|
||||
- Coverage % (requirements with >=1 task)
|
||||
- Ambiguity Count
|
||||
- Duplication Count
|
||||
- Critical Issues Count
|
||||
|
||||
### 7. Provide Next Actions
|
||||
|
||||
At end of report, output a concise Next Actions block:
|
||||
|
||||
- If CRITICAL issues exist: Recommend resolving before `/speckit.implement`
|
||||
- If only LOW/MEDIUM: User may proceed, but provide improvement suggestions
|
||||
- Provide explicit command suggestions: e.g., "Run /speckit.specify with refinement", "Run /speckit.plan to adjust architecture", "Manually edit tasks.md to add coverage for 'performance-metrics'"
|
||||
|
||||
### 8. Offer Remediation
|
||||
|
||||
Ask the user: "Would you like me to suggest concrete remediation edits for the top N issues?" (Do NOT apply them automatically.)
|
||||
|
||||
## Operating Principles
|
||||
|
||||
### Context Efficiency
|
||||
|
||||
- **Minimal high-signal tokens**: Focus on actionable findings, not exhaustive documentation
|
||||
- **Progressive disclosure**: Load artifacts incrementally; don't dump all content into analysis
|
||||
- **Token-efficient output**: Limit findings table to 50 rows; summarize overflow
|
||||
- **Deterministic results**: Rerunning without changes should produce consistent IDs and counts
|
||||
|
||||
### Analysis Guidelines
|
||||
|
||||
- **NEVER modify files** (this is read-only analysis)
|
||||
- **NEVER hallucinate missing sections** (if absent, report them accurately)
|
||||
- **Prioritize constitution violations** (these are always CRITICAL)
|
||||
- **Use examples over exhaustive rules** (cite specific instances, not generic patterns)
|
||||
- **Report zero issues gracefully** (emit success report with coverage statistics)
|
||||
|
||||
## Context
|
||||
|
||||
$ARGUMENTS
|
||||
294
nextflow/.claude/commands/speckit.checklist.md
Normal file
294
nextflow/.claude/commands/speckit.checklist.md
Normal file
@@ -0,0 +1,294 @@
|
||||
---
|
||||
description: Generate a custom checklist for the current feature based on user requirements.
|
||||
---
|
||||
|
||||
## Checklist Purpose: "Unit Tests for English"
|
||||
|
||||
**CRITICAL CONCEPT**: Checklists are **UNIT TESTS FOR REQUIREMENTS WRITING** - they validate the quality, clarity, and completeness of requirements in a given domain.
|
||||
|
||||
**NOT for verification/testing**:
|
||||
|
||||
- ❌ NOT "Verify the button clicks correctly"
|
||||
- ❌ NOT "Test error handling works"
|
||||
- ❌ NOT "Confirm the API returns 200"
|
||||
- ❌ NOT checking if code/implementation matches the spec
|
||||
|
||||
**FOR requirements quality validation**:
|
||||
|
||||
- ✅ "Are visual hierarchy requirements defined for all card types?" (completeness)
|
||||
- ✅ "Is 'prominent display' quantified with specific sizing/positioning?" (clarity)
|
||||
- ✅ "Are hover state requirements consistent across all interactive elements?" (consistency)
|
||||
- ✅ "Are accessibility requirements defined for keyboard navigation?" (coverage)
|
||||
- ✅ "Does the spec define what happens when logo image fails to load?" (edge cases)
|
||||
|
||||
**Metaphor**: If your spec is code written in English, the checklist is its unit test suite. You're testing whether the requirements are well-written, complete, unambiguous, and ready for implementation - NOT whether the implementation works.
|
||||
|
||||
## User Input
|
||||
|
||||
```text
|
||||
$ARGUMENTS
|
||||
```
|
||||
|
||||
You **MUST** consider the user input before proceeding (if not empty).
|
||||
|
||||
## Execution Steps
|
||||
|
||||
1. **Setup**: Run `.specify/scripts/bash/check-prerequisites.sh --json` from repo root and parse JSON for FEATURE_DIR and AVAILABLE_DOCS list.
|
||||
- All file paths must be absolute.
|
||||
- For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
||||
|
||||
2. **Clarify intent (dynamic)**: Derive up to THREE initial contextual clarifying questions (no pre-baked catalog). They MUST:
|
||||
- Be generated from the user's phrasing + extracted signals from spec/plan/tasks
|
||||
- Only ask about information that materially changes checklist content
|
||||
- Be skipped individually if already unambiguous in `$ARGUMENTS`
|
||||
- Prefer precision over breadth
|
||||
|
||||
Generation algorithm:
|
||||
1. Extract signals: feature domain keywords (e.g., auth, latency, UX, API), risk indicators ("critical", "must", "compliance"), stakeholder hints ("QA", "review", "security team"), and explicit deliverables ("a11y", "rollback", "contracts").
|
||||
2. Cluster signals into candidate focus areas (max 4) ranked by relevance.
|
||||
3. Identify probable audience & timing (author, reviewer, QA, release) if not explicit.
|
||||
4. Detect missing dimensions: scope breadth, depth/rigor, risk emphasis, exclusion boundaries, measurable acceptance criteria.
|
||||
5. Formulate questions chosen from these archetypes:
|
||||
- Scope refinement (e.g., "Should this include integration touchpoints with X and Y or stay limited to local module correctness?")
|
||||
- Risk prioritization (e.g., "Which of these potential risk areas should receive mandatory gating checks?")
|
||||
- Depth calibration (e.g., "Is this a lightweight pre-commit sanity list or a formal release gate?")
|
||||
- Audience framing (e.g., "Will this be used by the author only or peers during PR review?")
|
||||
- Boundary exclusion (e.g., "Should we explicitly exclude performance tuning items this round?")
|
||||
- Scenario class gap (e.g., "No recovery flows detected—are rollback / partial failure paths in scope?")
|
||||
|
||||
Question formatting rules:
|
||||
- If presenting options, generate a compact table with columns: Option | Candidate | Why It Matters
|
||||
- Limit to A–E options maximum; omit table if a free-form answer is clearer
|
||||
- Never ask the user to restate what they already said
|
||||
- Avoid speculative categories (no hallucination). If uncertain, ask explicitly: "Confirm whether X belongs in scope."
|
||||
|
||||
Defaults when interaction impossible:
|
||||
- Depth: Standard
|
||||
- Audience: Reviewer (PR) if code-related; Author otherwise
|
||||
- Focus: Top 2 relevance clusters
|
||||
|
||||
Output the questions (label Q1/Q2/Q3). After answers: if ≥2 scenario classes (Alternate / Exception / Recovery / Non-Functional domain) remain unclear, you MAY ask up to TWO more targeted follow‑ups (Q4/Q5) with a one-line justification each (e.g., "Unresolved recovery path risk"). Do not exceed five total questions. Skip escalation if user explicitly declines more.
|
||||
|
||||
3. **Understand user request**: Combine `$ARGUMENTS` + clarifying answers:
|
||||
- Derive checklist theme (e.g., security, review, deploy, ux)
|
||||
- Consolidate explicit must-have items mentioned by user
|
||||
- Map focus selections to category scaffolding
|
||||
- Infer any missing context from spec/plan/tasks (do NOT hallucinate)
|
||||
|
||||
4. **Load feature context**: Read from FEATURE_DIR:
|
||||
- spec.md: Feature requirements and scope
|
||||
- plan.md (if exists): Technical details, dependencies
|
||||
- tasks.md (if exists): Implementation tasks
|
||||
|
||||
**Context Loading Strategy**:
|
||||
- Load only necessary portions relevant to active focus areas (avoid full-file dumping)
|
||||
- Prefer summarizing long sections into concise scenario/requirement bullets
|
||||
- Use progressive disclosure: add follow-on retrieval only if gaps detected
|
||||
- If source docs are large, generate interim summary items instead of embedding raw text
|
||||
|
||||
5. **Generate checklist** - Create "Unit Tests for Requirements":
|
||||
- Create `FEATURE_DIR/checklists/` directory if it doesn't exist
|
||||
- Generate unique checklist filename:
|
||||
- Use short, descriptive name based on domain (e.g., `ux.md`, `api.md`, `security.md`)
|
||||
- Format: `[domain].md`
|
||||
- If file exists, append to existing file
|
||||
- Number items sequentially starting from CHK001
|
||||
- Each `/speckit.checklist` run creates a NEW file (never overwrites existing checklists)
|
||||
|
||||
**CORE PRINCIPLE - Test the Requirements, Not the Implementation**:
|
||||
Every checklist item MUST evaluate the REQUIREMENTS THEMSELVES for:
|
||||
- **Completeness**: Are all necessary requirements present?
|
||||
- **Clarity**: Are requirements unambiguous and specific?
|
||||
- **Consistency**: Do requirements align with each other?
|
||||
- **Measurability**: Can requirements be objectively verified?
|
||||
- **Coverage**: Are all scenarios/edge cases addressed?
|
||||
|
||||
**Category Structure** - Group items by requirement quality dimensions:
|
||||
- **Requirement Completeness** (Are all necessary requirements documented?)
|
||||
- **Requirement Clarity** (Are requirements specific and unambiguous?)
|
||||
- **Requirement Consistency** (Do requirements align without conflicts?)
|
||||
- **Acceptance Criteria Quality** (Are success criteria measurable?)
|
||||
- **Scenario Coverage** (Are all flows/cases addressed?)
|
||||
- **Edge Case Coverage** (Are boundary conditions defined?)
|
||||
- **Non-Functional Requirements** (Performance, Security, Accessibility, etc. - are they specified?)
|
||||
- **Dependencies & Assumptions** (Are they documented and validated?)
|
||||
- **Ambiguities & Conflicts** (What needs clarification?)
|
||||
|
||||
**HOW TO WRITE CHECKLIST ITEMS - "Unit Tests for English"**:
|
||||
|
||||
❌ **WRONG** (Testing implementation):
|
||||
- "Verify landing page displays 3 episode cards"
|
||||
- "Test hover states work on desktop"
|
||||
- "Confirm logo click navigates home"
|
||||
|
||||
✅ **CORRECT** (Testing requirements quality):
|
||||
- "Are the exact number and layout of featured episodes specified?" [Completeness]
|
||||
- "Is 'prominent display' quantified with specific sizing/positioning?" [Clarity]
|
||||
- "Are hover state requirements consistent across all interactive elements?" [Consistency]
|
||||
- "Are keyboard navigation requirements defined for all interactive UI?" [Coverage]
|
||||
- "Is the fallback behavior specified when logo image fails to load?" [Edge Cases]
|
||||
- "Are loading states defined for asynchronous episode data?" [Completeness]
|
||||
- "Does the spec define visual hierarchy for competing UI elements?" [Clarity]
|
||||
|
||||
**ITEM STRUCTURE**:
|
||||
Each item should follow this pattern:
|
||||
- Question format asking about requirement quality
|
||||
- Focus on what's WRITTEN (or not written) in the spec/plan
|
||||
- Include quality dimension in brackets [Completeness/Clarity/Consistency/etc.]
|
||||
- Reference spec section `[Spec §X.Y]` when checking existing requirements
|
||||
- Use `[Gap]` marker when checking for missing requirements
|
||||
|
||||
**EXAMPLES BY QUALITY DIMENSION**:
|
||||
|
||||
Completeness:
|
||||
- "Are error handling requirements defined for all API failure modes? [Gap]"
|
||||
- "Are accessibility requirements specified for all interactive elements? [Completeness]"
|
||||
- "Are mobile breakpoint requirements defined for responsive layouts? [Gap]"
|
||||
|
||||
Clarity:
|
||||
- "Is 'fast loading' quantified with specific timing thresholds? [Clarity, Spec §NFR-2]"
|
||||
- "Are 'related episodes' selection criteria explicitly defined? [Clarity, Spec §FR-5]"
|
||||
- "Is 'prominent' defined with measurable visual properties? [Ambiguity, Spec §FR-4]"
|
||||
|
||||
Consistency:
|
||||
- "Do navigation requirements align across all pages? [Consistency, Spec §FR-10]"
|
||||
- "Are card component requirements consistent between landing and detail pages? [Consistency]"
|
||||
|
||||
Coverage:
|
||||
- "Are requirements defined for zero-state scenarios (no episodes)? [Coverage, Edge Case]"
|
||||
- "Are concurrent user interaction scenarios addressed? [Coverage, Gap]"
|
||||
- "Are requirements specified for partial data loading failures? [Coverage, Exception Flow]"
|
||||
|
||||
Measurability:
|
||||
- "Are visual hierarchy requirements measurable/testable? [Acceptance Criteria, Spec §FR-1]"
|
||||
- "Can 'balanced visual weight' be objectively verified? [Measurability, Spec §FR-2]"
|
||||
|
||||
**Scenario Classification & Coverage** (Requirements Quality Focus):
|
||||
- Check if requirements exist for: Primary, Alternate, Exception/Error, Recovery, Non-Functional scenarios
|
||||
- For each scenario class, ask: "Are [scenario type] requirements complete, clear, and consistent?"
|
||||
- If scenario class missing: "Are [scenario type] requirements intentionally excluded or missing? [Gap]"
|
||||
- Include resilience/rollback when state mutation occurs: "Are rollback requirements defined for migration failures? [Gap]"
|
||||
|
||||
**Traceability Requirements**:
|
||||
- MINIMUM: ≥80% of items MUST include at least one traceability reference
|
||||
- Each item should reference: spec section `[Spec §X.Y]`, or use markers: `[Gap]`, `[Ambiguity]`, `[Conflict]`, `[Assumption]`
|
||||
- If no ID system exists: "Is a requirement & acceptance criteria ID scheme established? [Traceability]"
|
||||
|
||||
**Surface & Resolve Issues** (Requirements Quality Problems):
|
||||
Ask questions about the requirements themselves:
|
||||
- Ambiguities: "Is the term 'fast' quantified with specific metrics? [Ambiguity, Spec §NFR-1]"
|
||||
- Conflicts: "Do navigation requirements conflict between §FR-10 and §FR-10a? [Conflict]"
|
||||
- Assumptions: "Is the assumption of 'always available podcast API' validated? [Assumption]"
|
||||
- Dependencies: "Are external podcast API requirements documented? [Dependency, Gap]"
|
||||
- Missing definitions: "Is 'visual hierarchy' defined with measurable criteria? [Gap]"
|
||||
|
||||
**Content Consolidation**:
|
||||
- Soft cap: If raw candidate items > 40, prioritize by risk/impact
|
||||
- Merge near-duplicates checking the same requirement aspect
|
||||
- If >5 low-impact edge cases, create one item: "Are edge cases X, Y, Z addressed in requirements? [Coverage]"
|
||||
|
||||
**🚫 ABSOLUTELY PROHIBITED** - These make it an implementation test, not a requirements test:
|
||||
- ❌ Any item starting with "Verify", "Test", "Confirm", "Check" + implementation behavior
|
||||
- ❌ References to code execution, user actions, system behavior
|
||||
- ❌ "Displays correctly", "works properly", "functions as expected"
|
||||
- ❌ "Click", "navigate", "render", "load", "execute"
|
||||
- ❌ Test cases, test plans, QA procedures
|
||||
- ❌ Implementation details (frameworks, APIs, algorithms)
|
||||
|
||||
**✅ REQUIRED PATTERNS** - These test requirements quality:
|
||||
- ✅ "Are [requirement type] defined/specified/documented for [scenario]?"
|
||||
- ✅ "Is [vague term] quantified/clarified with specific criteria?"
|
||||
- ✅ "Are requirements consistent between [section A] and [section B]?"
|
||||
- ✅ "Can [requirement] be objectively measured/verified?"
|
||||
- ✅ "Are [edge cases/scenarios] addressed in requirements?"
|
||||
- ✅ "Does the spec define [missing aspect]?"
|
||||
|
||||
6. **Structure Reference**: Generate the checklist following the canonical template in `.specify/templates/checklist-template.md` for title, meta section, category headings, and ID formatting. If template is unavailable, use: H1 title, purpose/created meta lines, `##` category sections containing `- [ ] CHK### <requirement item>` lines with globally incrementing IDs starting at CHK001.
|
||||
|
||||
7. **Report**: Output full path to created checklist, item count, and remind user that each run creates a new file. Summarize:
|
||||
- Focus areas selected
|
||||
- Depth level
|
||||
- Actor/timing
|
||||
- Any explicit user-specified must-have items incorporated
|
||||
|
||||
**Important**: Each `/speckit.checklist` command invocation creates a checklist file using short, descriptive names unless file already exists. This allows:
|
||||
|
||||
- Multiple checklists of different types (e.g., `ux.md`, `test.md`, `security.md`)
|
||||
- Simple, memorable filenames that indicate checklist purpose
|
||||
- Easy identification and navigation in the `checklists/` folder
|
||||
|
||||
To avoid clutter, use descriptive types and clean up obsolete checklists when done.
|
||||
|
||||
## Example Checklist Types & Sample Items
|
||||
|
||||
**UX Requirements Quality:** `ux.md`
|
||||
|
||||
Sample items (testing the requirements, NOT the implementation):
|
||||
|
||||
- "Are visual hierarchy requirements defined with measurable criteria? [Clarity, Spec §FR-1]"
|
||||
- "Is the number and positioning of UI elements explicitly specified? [Completeness, Spec §FR-1]"
|
||||
- "Are interaction state requirements (hover, focus, active) consistently defined? [Consistency]"
|
||||
- "Are accessibility requirements specified for all interactive elements? [Coverage, Gap]"
|
||||
- "Is fallback behavior defined when images fail to load? [Edge Case, Gap]"
|
||||
- "Can 'prominent display' be objectively measured? [Measurability, Spec §FR-4]"
|
||||
|
||||
**API Requirements Quality:** `api.md`
|
||||
|
||||
Sample items:
|
||||
|
||||
- "Are error response formats specified for all failure scenarios? [Completeness]"
|
||||
- "Are rate limiting requirements quantified with specific thresholds? [Clarity]"
|
||||
- "Are authentication requirements consistent across all endpoints? [Consistency]"
|
||||
- "Are retry/timeout requirements defined for external dependencies? [Coverage, Gap]"
|
||||
- "Is versioning strategy documented in requirements? [Gap]"
|
||||
|
||||
**Performance Requirements Quality:** `performance.md`
|
||||
|
||||
Sample items:
|
||||
|
||||
- "Are performance requirements quantified with specific metrics? [Clarity]"
|
||||
- "Are performance targets defined for all critical user journeys? [Coverage]"
|
||||
- "Are performance requirements under different load conditions specified? [Completeness]"
|
||||
- "Can performance requirements be objectively measured? [Measurability]"
|
||||
- "Are degradation requirements defined for high-load scenarios? [Edge Case, Gap]"
|
||||
|
||||
**Security Requirements Quality:** `security.md`
|
||||
|
||||
Sample items:
|
||||
|
||||
- "Are authentication requirements specified for all protected resources? [Coverage]"
|
||||
- "Are data protection requirements defined for sensitive information? [Completeness]"
|
||||
- "Is the threat model documented and requirements aligned to it? [Traceability]"
|
||||
- "Are security requirements consistent with compliance obligations? [Consistency]"
|
||||
- "Are security failure/breach response requirements defined? [Gap, Exception Flow]"
|
||||
|
||||
## Anti-Examples: What NOT To Do
|
||||
|
||||
**❌ WRONG - These test implementation, not requirements:**
|
||||
|
||||
```markdown
|
||||
- [ ] CHK001 - Verify landing page displays 3 episode cards [Spec §FR-001]
|
||||
- [ ] CHK002 - Test hover states work correctly on desktop [Spec §FR-003]
|
||||
- [ ] CHK003 - Confirm logo click navigates to home page [Spec §FR-010]
|
||||
- [ ] CHK004 - Check that related episodes section shows 3-5 items [Spec §FR-005]
|
||||
```
|
||||
|
||||
**✅ CORRECT - These test requirements quality:**
|
||||
|
||||
```markdown
|
||||
- [ ] CHK001 - Are the number and layout of featured episodes explicitly specified? [Completeness, Spec §FR-001]
|
||||
- [ ] CHK002 - Are hover state requirements consistently defined for all interactive elements? [Consistency, Spec §FR-003]
|
||||
- [ ] CHK003 - Are navigation requirements clear for all clickable brand elements? [Clarity, Spec §FR-010]
|
||||
- [ ] CHK004 - Is the selection criteria for related episodes documented? [Gap, Spec §FR-005]
|
||||
- [ ] CHK005 - Are loading state requirements defined for asynchronous episode data? [Gap]
|
||||
- [ ] CHK006 - Can "visual hierarchy" requirements be objectively measured? [Measurability, Spec §FR-001]
|
||||
```
|
||||
|
||||
**Key Differences:**
|
||||
|
||||
- Wrong: Tests if the system works correctly
|
||||
- Correct: Tests if the requirements are written correctly
|
||||
- Wrong: Verification of behavior
|
||||
- Correct: Validation of requirement quality
|
||||
- Wrong: "Does it do X?"
|
||||
- Correct: "Is X clearly specified?"
|
||||
181
nextflow/.claude/commands/speckit.clarify.md
Normal file
181
nextflow/.claude/commands/speckit.clarify.md
Normal file
@@ -0,0 +1,181 @@
|
||||
---
|
||||
description: Identify underspecified areas in the current feature spec by asking up to 5 highly targeted clarification questions and encoding answers back into the spec.
|
||||
handoffs:
|
||||
- label: Build Technical Plan
|
||||
agent: speckit.plan
|
||||
prompt: Create a plan for the spec. I am building with...
|
||||
---
|
||||
|
||||
## User Input
|
||||
|
||||
```text
|
||||
$ARGUMENTS
|
||||
```
|
||||
|
||||
You **MUST** consider the user input before proceeding (if not empty).
|
||||
|
||||
## Outline
|
||||
|
||||
Goal: Detect and reduce ambiguity or missing decision points in the active feature specification and record the clarifications directly in the spec file.
|
||||
|
||||
Note: This clarification workflow is expected to run (and be completed) BEFORE invoking `/speckit.plan`. If the user explicitly states they are skipping clarification (e.g., exploratory spike), you may proceed, but must warn that downstream rework risk increases.
|
||||
|
||||
Execution steps:
|
||||
|
||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --paths-only` from repo root **once** (combined `--json --paths-only` mode / `-Json -PathsOnly`). Parse minimal JSON payload fields:
|
||||
- `FEATURE_DIR`
|
||||
- `FEATURE_SPEC`
|
||||
- (Optionally capture `IMPL_PLAN`, `TASKS` for future chained flows.)
|
||||
- If JSON parsing fails, abort and instruct user to re-run `/speckit.specify` or verify feature branch environment.
|
||||
- For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
||||
|
||||
2. Load the current spec file. Perform a structured ambiguity & coverage scan using this taxonomy. For each category, mark status: Clear / Partial / Missing. Produce an internal coverage map used for prioritization (do not output raw map unless no questions will be asked).
|
||||
|
||||
Functional Scope & Behavior:
|
||||
- Core user goals & success criteria
|
||||
- Explicit out-of-scope declarations
|
||||
- User roles / personas differentiation
|
||||
|
||||
Domain & Data Model:
|
||||
- Entities, attributes, relationships
|
||||
- Identity & uniqueness rules
|
||||
- Lifecycle/state transitions
|
||||
- Data volume / scale assumptions
|
||||
|
||||
Interaction & UX Flow:
|
||||
- Critical user journeys / sequences
|
||||
- Error/empty/loading states
|
||||
- Accessibility or localization notes
|
||||
|
||||
Non-Functional Quality Attributes:
|
||||
- Performance (latency, throughput targets)
|
||||
- Scalability (horizontal/vertical, limits)
|
||||
- Reliability & availability (uptime, recovery expectations)
|
||||
- Observability (logging, metrics, tracing signals)
|
||||
- Security & privacy (authN/Z, data protection, threat assumptions)
|
||||
- Compliance / regulatory constraints (if any)
|
||||
|
||||
Integration & External Dependencies:
|
||||
- External services/APIs and failure modes
|
||||
- Data import/export formats
|
||||
- Protocol/versioning assumptions
|
||||
|
||||
Edge Cases & Failure Handling:
|
||||
- Negative scenarios
|
||||
- Rate limiting / throttling
|
||||
- Conflict resolution (e.g., concurrent edits)
|
||||
|
||||
Constraints & Tradeoffs:
|
||||
- Technical constraints (language, storage, hosting)
|
||||
- Explicit tradeoffs or rejected alternatives
|
||||
|
||||
Terminology & Consistency:
|
||||
- Canonical glossary terms
|
||||
- Avoided synonyms / deprecated terms
|
||||
|
||||
Completion Signals:
|
||||
- Acceptance criteria testability
|
||||
- Measurable Definition of Done style indicators
|
||||
|
||||
Misc / Placeholders:
|
||||
- TODO markers / unresolved decisions
|
||||
- Ambiguous adjectives ("robust", "intuitive") lacking quantification
|
||||
|
||||
For each category with Partial or Missing status, add a candidate question opportunity unless:
|
||||
- Clarification would not materially change implementation or validation strategy
|
||||
- Information is better deferred to planning phase (note internally)
|
||||
|
||||
3. Generate (internally) a prioritized queue of candidate clarification questions (maximum 5). Do NOT output them all at once. Apply these constraints:
|
||||
- Maximum of 10 total questions across the whole session.
|
||||
- Each question must be answerable with EITHER:
|
||||
- A short multiple‑choice selection (2–5 distinct, mutually exclusive options), OR
|
||||
- A one-word / short‑phrase answer (explicitly constrain: "Answer in <=5 words").
|
||||
- Only include questions whose answers materially impact architecture, data modeling, task decomposition, test design, UX behavior, operational readiness, or compliance validation.
|
||||
- Ensure category coverage balance: attempt to cover the highest impact unresolved categories first; avoid asking two low-impact questions when a single high-impact area (e.g., security posture) is unresolved.
|
||||
- Exclude questions already answered, trivial stylistic preferences, or plan-level execution details (unless blocking correctness).
|
||||
- Favor clarifications that reduce downstream rework risk or prevent misaligned acceptance tests.
|
||||
- If more than 5 categories remain unresolved, select the top 5 by (Impact * Uncertainty) heuristic.
|
||||
|
||||
4. Sequential questioning loop (interactive):
|
||||
- Present EXACTLY ONE question at a time.
|
||||
- For multiple‑choice questions:
|
||||
- **Analyze all options** and determine the **most suitable option** based on:
|
||||
- Best practices for the project type
|
||||
- Common patterns in similar implementations
|
||||
- Risk reduction (security, performance, maintainability)
|
||||
- Alignment with any explicit project goals or constraints visible in the spec
|
||||
- Present your **recommended option prominently** at the top with clear reasoning (1-2 sentences explaining why this is the best choice).
|
||||
- Format as: `**Recommended:** Option [X] - <reasoning>`
|
||||
- Then render all options as a Markdown table:
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| A | <Option A description> |
|
||||
| B | <Option B description> |
|
||||
| C | <Option C description> (add D/E as needed up to 5) |
|
||||
| Short | Provide a different short answer (<=5 words) (Include only if free-form alternative is appropriate) |
|
||||
|
||||
- After the table, add: `You can reply with the option letter (e.g., "A"), accept the recommendation by saying "yes" or "recommended", or provide your own short answer.`
|
||||
- For short‑answer style (no meaningful discrete options):
|
||||
- Provide your **suggested answer** based on best practices and context.
|
||||
- Format as: `**Suggested:** <your proposed answer> - <brief reasoning>`
|
||||
- Then output: `Format: Short answer (<=5 words). You can accept the suggestion by saying "yes" or "suggested", or provide your own answer.`
|
||||
- After the user answers:
|
||||
- If the user replies with "yes", "recommended", or "suggested", use your previously stated recommendation/suggestion as the answer.
|
||||
- Otherwise, validate the answer maps to one option or fits the <=5 word constraint.
|
||||
- If ambiguous, ask for a quick disambiguation (count still belongs to same question; do not advance).
|
||||
- Once satisfactory, record it in working memory (do not yet write to disk) and move to the next queued question.
|
||||
- Stop asking further questions when:
|
||||
- All critical ambiguities resolved early (remaining queued items become unnecessary), OR
|
||||
- User signals completion ("done", "good", "no more"), OR
|
||||
- You reach 5 asked questions.
|
||||
- Never reveal future queued questions in advance.
|
||||
- If no valid questions exist at start, immediately report no critical ambiguities.
|
||||
|
||||
5. Integration after EACH accepted answer (incremental update approach):
|
||||
- Maintain in-memory representation of the spec (loaded once at start) plus the raw file contents.
|
||||
- For the first integrated answer in this session:
|
||||
- Ensure a `## Clarifications` section exists (create it just after the highest-level contextual/overview section per the spec template if missing).
|
||||
- Under it, create (if not present) a `### Session YYYY-MM-DD` subheading for today.
|
||||
- Append a bullet line immediately after acceptance: `- Q: <question> → A: <final answer>`.
|
||||
- Then immediately apply the clarification to the most appropriate section(s):
|
||||
- Functional ambiguity → Update or add a bullet in Functional Requirements.
|
||||
- User interaction / actor distinction → Update User Stories or Actors subsection (if present) with clarified role, constraint, or scenario.
|
||||
- Data shape / entities → Update Data Model (add fields, types, relationships) preserving ordering; note added constraints succinctly.
|
||||
- Non-functional constraint → Add/modify measurable criteria in Non-Functional / Quality Attributes section (convert vague adjective to metric or explicit target).
|
||||
- Edge case / negative flow → Add a new bullet under Edge Cases / Error Handling (or create such subsection if template provides placeholder for it).
|
||||
- Terminology conflict → Normalize term across spec; retain original only if necessary by adding `(formerly referred to as "X")` once.
|
||||
- If the clarification invalidates an earlier ambiguous statement, replace that statement instead of duplicating; leave no obsolete contradictory text.
|
||||
- Save the spec file AFTER each integration to minimize risk of context loss (atomic overwrite).
|
||||
- Preserve formatting: do not reorder unrelated sections; keep heading hierarchy intact.
|
||||
- Keep each inserted clarification minimal and testable (avoid narrative drift).
|
||||
|
||||
6. Validation (performed after EACH write plus final pass):
|
||||
- Clarifications session contains exactly one bullet per accepted answer (no duplicates).
|
||||
- Total asked (accepted) questions ≤ 5.
|
||||
- Updated sections contain no lingering vague placeholders the new answer was meant to resolve.
|
||||
- No contradictory earlier statement remains (scan for now-invalid alternative choices removed).
|
||||
- Markdown structure valid; only allowed new headings: `## Clarifications`, `### Session YYYY-MM-DD`.
|
||||
- Terminology consistency: same canonical term used across all updated sections.
|
||||
|
||||
7. Write the updated spec back to `FEATURE_SPEC`.
|
||||
|
||||
8. Report completion (after questioning loop ends or early termination):
|
||||
- Number of questions asked & answered.
|
||||
- Path to updated spec.
|
||||
- Sections touched (list names).
|
||||
- Coverage summary table listing each taxonomy category with Status: Resolved (was Partial/Missing and addressed), Deferred (exceeds question quota or better suited for planning), Clear (already sufficient), Outstanding (still Partial/Missing but low impact).
|
||||
- If any Outstanding or Deferred remain, recommend whether to proceed to `/speckit.plan` or run `/speckit.clarify` again later post-plan.
|
||||
- Suggested next command.
|
||||
|
||||
Behavior rules:
|
||||
|
||||
- If no meaningful ambiguities found (or all potential questions would be low-impact), respond: "No critical ambiguities detected worth formal clarification." and suggest proceeding.
|
||||
- If spec file missing, instruct user to run `/speckit.specify` first (do not create a new spec here).
|
||||
- Never exceed 5 total asked questions (clarification retries for a single question do not count as new questions).
|
||||
- Avoid speculative tech stack questions unless the absence blocks functional clarity.
|
||||
- Respect user early termination signals ("stop", "done", "proceed").
|
||||
- If no questions asked due to full coverage, output a compact coverage summary (all categories Clear) then suggest advancing.
|
||||
- If quota reached with unresolved high-impact categories remaining, explicitly flag them under Deferred with rationale.
|
||||
|
||||
Context for prioritization: $ARGUMENTS
|
||||
82
nextflow/.claude/commands/speckit.constitution.md
Normal file
82
nextflow/.claude/commands/speckit.constitution.md
Normal file
@@ -0,0 +1,82 @@
|
||||
---
|
||||
description: Create or update the project constitution from interactive or provided principle inputs, ensuring all dependent templates stay in sync.
|
||||
handoffs:
|
||||
- label: Build Specification
|
||||
agent: speckit.specify
|
||||
prompt: Implement the feature specification based on the updated constitution. I want to build...
|
||||
---
|
||||
|
||||
## User Input
|
||||
|
||||
```text
|
||||
$ARGUMENTS
|
||||
```
|
||||
|
||||
You **MUST** consider the user input before proceeding (if not empty).
|
||||
|
||||
## Outline
|
||||
|
||||
You are updating the project constitution at `.specify/memory/constitution.md`. This file is a TEMPLATE containing placeholder tokens in square brackets (e.g. `[PROJECT_NAME]`, `[PRINCIPLE_1_NAME]`). Your job is to (a) collect/derive concrete values, (b) fill the template precisely, and (c) propagate any amendments across dependent artifacts.
|
||||
|
||||
Follow this execution flow:
|
||||
|
||||
1. Load the existing constitution template at `.specify/memory/constitution.md`.
|
||||
- Identify every placeholder token of the form `[ALL_CAPS_IDENTIFIER]`.
|
||||
**IMPORTANT**: The user might require less or more principles than the ones used in the template. If a number is specified, respect that - follow the general template. You will update the doc accordingly.
|
||||
|
||||
2. Collect/derive values for placeholders:
|
||||
- If user input (conversation) supplies a value, use it.
|
||||
- Otherwise infer from existing repo context (README, docs, prior constitution versions if embedded).
|
||||
- For governance dates: `RATIFICATION_DATE` is the original adoption date (if unknown ask or mark TODO), `LAST_AMENDED_DATE` is today if changes are made, otherwise keep previous.
|
||||
- `CONSTITUTION_VERSION` must increment according to semantic versioning rules:
|
||||
- MAJOR: Backward incompatible governance/principle removals or redefinitions.
|
||||
- MINOR: New principle/section added or materially expanded guidance.
|
||||
- PATCH: Clarifications, wording, typo fixes, non-semantic refinements.
|
||||
- If version bump type ambiguous, propose reasoning before finalizing.
|
||||
|
||||
3. Draft the updated constitution content:
|
||||
- Replace every placeholder with concrete text (no bracketed tokens left except intentionally retained template slots that the project has chosen not to define yet—explicitly justify any left).
|
||||
- Preserve heading hierarchy and comments can be removed once replaced unless they still add clarifying guidance.
|
||||
- Ensure each Principle section: succinct name line, paragraph (or bullet list) capturing non‑negotiable rules, explicit rationale if not obvious.
|
||||
- Ensure Governance section lists amendment procedure, versioning policy, and compliance review expectations.
|
||||
|
||||
4. Consistency propagation checklist (convert prior checklist into active validations):
|
||||
- Read `.specify/templates/plan-template.md` and ensure any "Constitution Check" or rules align with updated principles.
|
||||
- Read `.specify/templates/spec-template.md` for scope/requirements alignment—update if constitution adds/removes mandatory sections or constraints.
|
||||
- Read `.specify/templates/tasks-template.md` and ensure task categorization reflects new or removed principle-driven task types (e.g., observability, versioning, testing discipline).
|
||||
- Read each command file in `.specify/templates/commands/*.md` (including this one) to verify no outdated references (agent-specific names like CLAUDE only) remain when generic guidance is required.
|
||||
- Read any runtime guidance docs (e.g., `README.md`, `docs/quickstart.md`, or agent-specific guidance files if present). Update references to principles changed.
|
||||
|
||||
5. Produce a Sync Impact Report (prepend as an HTML comment at top of the constitution file after update):
|
||||
- Version change: old → new
|
||||
- List of modified principles (old title → new title if renamed)
|
||||
- Added sections
|
||||
- Removed sections
|
||||
- Templates requiring updates (✅ updated / ⚠ pending) with file paths
|
||||
- Follow-up TODOs if any placeholders intentionally deferred.
|
||||
|
||||
6. Validation before final output:
|
||||
- No remaining unexplained bracket tokens.
|
||||
- Version line matches report.
|
||||
- Dates ISO format YYYY-MM-DD.
|
||||
- Principles are declarative, testable, and free of vague language ("should" → replace with MUST/SHOULD rationale where appropriate).
|
||||
|
||||
7. Write the completed constitution back to `.specify/memory/constitution.md` (overwrite).
|
||||
|
||||
8. Output a final summary to the user with:
|
||||
- New version and bump rationale.
|
||||
- Any files flagged for manual follow-up.
|
||||
- Suggested commit message (e.g., `docs: amend constitution to vX.Y.Z (principle additions + governance update)`).
|
||||
|
||||
Formatting & Style Requirements:
|
||||
|
||||
- Use Markdown headings exactly as in the template (do not demote/promote levels).
|
||||
- Wrap long rationale lines to keep readability (<100 chars ideally) but do not hard enforce with awkward breaks.
|
||||
- Keep a single blank line between sections.
|
||||
- Avoid trailing whitespace.
|
||||
|
||||
If the user supplies partial updates (e.g., only one principle revision), still perform validation and version decision steps.
|
||||
|
||||
If critical info missing (e.g., ratification date truly unknown), insert `TODO(<FIELD_NAME>): explanation` and include in the Sync Impact Report under deferred items.
|
||||
|
||||
Do not create a new template; always operate on the existing `.specify/memory/constitution.md` file.
|
||||
135
nextflow/.claude/commands/speckit.implement.md
Normal file
135
nextflow/.claude/commands/speckit.implement.md
Normal file
@@ -0,0 +1,135 @@
|
||||
---
|
||||
description: Execute the implementation plan by processing and executing all tasks defined in tasks.md
|
||||
---
|
||||
|
||||
## User Input
|
||||
|
||||
```text
|
||||
$ARGUMENTS
|
||||
```
|
||||
|
||||
You **MUST** consider the user input before proceeding (if not empty).
|
||||
|
||||
## Outline
|
||||
|
||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
||||
|
||||
2. **Check checklists status** (if FEATURE_DIR/checklists/ exists):
|
||||
- Scan all checklist files in the checklists/ directory
|
||||
- For each checklist, count:
|
||||
- Total items: All lines matching `- [ ]` or `- [X]` or `- [x]`
|
||||
- Completed items: Lines matching `- [X]` or `- [x]`
|
||||
- Incomplete items: Lines matching `- [ ]`
|
||||
- Create a status table:
|
||||
|
||||
```text
|
||||
| Checklist | Total | Completed | Incomplete | Status |
|
||||
|-----------|-------|-----------|------------|--------|
|
||||
| ux.md | 12 | 12 | 0 | ✓ PASS |
|
||||
| test.md | 8 | 5 | 3 | ✗ FAIL |
|
||||
| security.md | 6 | 6 | 0 | ✓ PASS |
|
||||
```
|
||||
|
||||
- Calculate overall status:
|
||||
- **PASS**: All checklists have 0 incomplete items
|
||||
- **FAIL**: One or more checklists have incomplete items
|
||||
|
||||
- **If any checklist is incomplete**:
|
||||
- Display the table with incomplete item counts
|
||||
- **STOP** and ask: "Some checklists are incomplete. Do you want to proceed with implementation anyway? (yes/no)"
|
||||
- Wait for user response before continuing
|
||||
- If user says "no" or "wait" or "stop", halt execution
|
||||
- If user says "yes" or "proceed" or "continue", proceed to step 3
|
||||
|
||||
- **If all checklists are complete**:
|
||||
- Display the table showing all checklists passed
|
||||
- Automatically proceed to step 3
|
||||
|
||||
3. Load and analyze the implementation context:
|
||||
- **REQUIRED**: Read tasks.md for the complete task list and execution plan
|
||||
- **REQUIRED**: Read plan.md for tech stack, architecture, and file structure
|
||||
- **IF EXISTS**: Read data-model.md for entities and relationships
|
||||
- **IF EXISTS**: Read contracts/ for API specifications and test requirements
|
||||
- **IF EXISTS**: Read research.md for technical decisions and constraints
|
||||
- **IF EXISTS**: Read quickstart.md for integration scenarios
|
||||
|
||||
4. **Project Setup Verification**:
|
||||
- **REQUIRED**: Create/verify ignore files based on actual project setup:
|
||||
|
||||
**Detection & Creation Logic**:
|
||||
- Check if the following command succeeds to determine if the repository is a git repo (create/verify .gitignore if so):
|
||||
|
||||
```sh
|
||||
git rev-parse --git-dir 2>/dev/null
|
||||
```
|
||||
|
||||
- Check if Dockerfile* exists or Docker in plan.md → create/verify .dockerignore
|
||||
- Check if .eslintrc* exists → create/verify .eslintignore
|
||||
- Check if eslint.config.* exists → ensure the config's `ignores` entries cover required patterns
|
||||
- Check if .prettierrc* exists → create/verify .prettierignore
|
||||
- Check if .npmrc or package.json exists → create/verify .npmignore (if publishing)
|
||||
- Check if terraform files (*.tf) exist → create/verify .terraformignore
|
||||
- Check if .helmignore needed (helm charts present) → create/verify .helmignore
|
||||
|
||||
**If ignore file already exists**: Verify it contains essential patterns, append missing critical patterns only
|
||||
**If ignore file missing**: Create with full pattern set for detected technology
|
||||
|
||||
**Common Patterns by Technology** (from plan.md tech stack):
|
||||
- **Node.js/JavaScript/TypeScript**: `node_modules/`, `dist/`, `build/`, `*.log`, `.env*`
|
||||
- **Python**: `__pycache__/`, `*.pyc`, `.venv/`, `venv/`, `dist/`, `*.egg-info/`
|
||||
- **Java**: `target/`, `*.class`, `*.jar`, `.gradle/`, `build/`
|
||||
- **C#/.NET**: `bin/`, `obj/`, `*.user`, `*.suo`, `packages/`
|
||||
- **Go**: `*.exe`, `*.test`, `vendor/`, `*.out`
|
||||
- **Ruby**: `.bundle/`, `log/`, `tmp/`, `*.gem`, `vendor/bundle/`
|
||||
- **PHP**: `vendor/`, `*.log`, `*.cache`, `*.env`
|
||||
- **Rust**: `target/`, `debug/`, `release/`, `*.rs.bk`, `*.rlib`, `*.prof*`, `.idea/`, `*.log`, `.env*`
|
||||
- **Kotlin**: `build/`, `out/`, `.gradle/`, `.idea/`, `*.class`, `*.jar`, `*.iml`, `*.log`, `.env*`
|
||||
- **C++**: `build/`, `bin/`, `obj/`, `out/`, `*.o`, `*.so`, `*.a`, `*.exe`, `*.dll`, `.idea/`, `*.log`, `.env*`
|
||||
- **C**: `build/`, `bin/`, `obj/`, `out/`, `*.o`, `*.a`, `*.so`, `*.exe`, `Makefile`, `config.log`, `.idea/`, `*.log`, `.env*`
|
||||
- **Swift**: `.build/`, `DerivedData/`, `*.swiftpm/`, `Packages/`
|
||||
- **R**: `.Rproj.user/`, `.Rhistory`, `.RData`, `.Ruserdata`, `*.Rproj`, `packrat/`, `renv/`
|
||||
- **Universal**: `.DS_Store`, `Thumbs.db`, `*.tmp`, `*.swp`, `.vscode/`, `.idea/`
|
||||
|
||||
**Tool-Specific Patterns**:
|
||||
- **Docker**: `node_modules/`, `.git/`, `Dockerfile*`, `.dockerignore`, `*.log*`, `.env*`, `coverage/`
|
||||
- **ESLint**: `node_modules/`, `dist/`, `build/`, `coverage/`, `*.min.js`
|
||||
- **Prettier**: `node_modules/`, `dist/`, `build/`, `coverage/`, `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`
|
||||
- **Terraform**: `.terraform/`, `*.tfstate*`, `*.tfvars`, `.terraform.lock.hcl`
|
||||
- **Kubernetes/k8s**: `*.secret.yaml`, `secrets/`, `.kube/`, `kubeconfig*`, `*.key`, `*.crt`
|
||||
|
||||
5. Parse tasks.md structure and extract:
|
||||
- **Task phases**: Setup, Tests, Core, Integration, Polish
|
||||
- **Task dependencies**: Sequential vs parallel execution rules
|
||||
- **Task details**: ID, description, file paths, parallel markers [P]
|
||||
- **Execution flow**: Order and dependency requirements
|
||||
|
||||
6. Execute implementation following the task plan:
|
||||
- **Phase-by-phase execution**: Complete each phase before moving to the next
|
||||
- **Respect dependencies**: Run sequential tasks in order, parallel tasks [P] can run together
|
||||
- **Follow TDD approach**: Execute test tasks before their corresponding implementation tasks
|
||||
- **File-based coordination**: Tasks affecting the same files must run sequentially
|
||||
- **Validation checkpoints**: Verify each phase completion before proceeding
|
||||
|
||||
7. Implementation execution rules:
|
||||
- **Setup first**: Initialize project structure, dependencies, configuration
|
||||
- **Tests before code**: If you need to write tests for contracts, entities, and integration scenarios
|
||||
- **Core development**: Implement models, services, CLI commands, endpoints
|
||||
- **Integration work**: Database connections, middleware, logging, external services
|
||||
- **Polish and validation**: Unit tests, performance optimization, documentation
|
||||
|
||||
8. Progress tracking and error handling:
|
||||
- Report progress after each completed task
|
||||
- Halt execution if any non-parallel task fails
|
||||
- For parallel tasks [P], continue with successful tasks, report failed ones
|
||||
- Provide clear error messages with context for debugging
|
||||
- Suggest next steps if implementation cannot proceed
|
||||
- **IMPORTANT** For completed tasks, make sure to mark the task off as [X] in the tasks file.
|
||||
|
||||
9. Completion validation:
|
||||
- Verify all required tasks are completed
|
||||
- Check that implemented features match the original specification
|
||||
- Validate that tests pass and coverage meets requirements
|
||||
- Confirm the implementation follows the technical plan
|
||||
- Report final status with summary of completed work
|
||||
|
||||
Note: This command assumes a complete task breakdown exists in tasks.md. If tasks are incomplete or missing, suggest running `/speckit.tasks` first to regenerate the task list.
|
||||
89
nextflow/.claude/commands/speckit.plan.md
Normal file
89
nextflow/.claude/commands/speckit.plan.md
Normal file
@@ -0,0 +1,89 @@
|
||||
---
|
||||
description: Execute the implementation planning workflow using the plan template to generate design artifacts.
|
||||
handoffs:
|
||||
- label: Create Tasks
|
||||
agent: speckit.tasks
|
||||
prompt: Break the plan into tasks
|
||||
send: true
|
||||
- label: Create Checklist
|
||||
agent: speckit.checklist
|
||||
prompt: Create a checklist for the following domain...
|
||||
---
|
||||
|
||||
## User Input
|
||||
|
||||
```text
|
||||
$ARGUMENTS
|
||||
```
|
||||
|
||||
You **MUST** consider the user input before proceeding (if not empty).
|
||||
|
||||
## Outline
|
||||
|
||||
1. **Setup**: Run `.specify/scripts/bash/setup-plan.sh --json` from repo root and parse JSON for FEATURE_SPEC, IMPL_PLAN, SPECS_DIR, BRANCH. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
||||
|
||||
2. **Load context**: Read FEATURE_SPEC and `.specify/memory/constitution.md`. Load IMPL_PLAN template (already copied).
|
||||
|
||||
3. **Execute plan workflow**: Follow the structure in IMPL_PLAN template to:
|
||||
- Fill Technical Context (mark unknowns as "NEEDS CLARIFICATION")
|
||||
- Fill Constitution Check section from constitution
|
||||
- Evaluate gates (ERROR if violations unjustified)
|
||||
- Phase 0: Generate research.md (resolve all NEEDS CLARIFICATION)
|
||||
- Phase 1: Generate data-model.md, contracts/, quickstart.md
|
||||
- Phase 1: Update agent context by running the agent script
|
||||
- Re-evaluate Constitution Check post-design
|
||||
|
||||
4. **Stop and report**: Command ends after Phase 2 planning. Report branch, IMPL_PLAN path, and generated artifacts.
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 0: Outline & Research
|
||||
|
||||
1. **Extract unknowns from Technical Context** above:
|
||||
- For each NEEDS CLARIFICATION → research task
|
||||
- For each dependency → best practices task
|
||||
- For each integration → patterns task
|
||||
|
||||
2. **Generate and dispatch research agents**:
|
||||
|
||||
```text
|
||||
For each unknown in Technical Context:
|
||||
Task: "Research {unknown} for {feature context}"
|
||||
For each technology choice:
|
||||
Task: "Find best practices for {tech} in {domain}"
|
||||
```
|
||||
|
||||
3. **Consolidate findings** in `research.md` using format:
|
||||
- Decision: [what was chosen]
|
||||
- Rationale: [why chosen]
|
||||
- Alternatives considered: [what else evaluated]
|
||||
|
||||
**Output**: research.md with all NEEDS CLARIFICATION resolved
|
||||
|
||||
### Phase 1: Design & Contracts
|
||||
|
||||
**Prerequisites:** `research.md` complete
|
||||
|
||||
1. **Extract entities from feature spec** → `data-model.md`:
|
||||
- Entity name, fields, relationships
|
||||
- Validation rules from requirements
|
||||
- State transitions if applicable
|
||||
|
||||
2. **Generate API contracts** from functional requirements:
|
||||
- For each user action → endpoint
|
||||
- Use standard REST/GraphQL patterns
|
||||
- Output OpenAPI/GraphQL schema to `/contracts/`
|
||||
|
||||
3. **Agent context update**:
|
||||
- Run `.specify/scripts/bash/update-agent-context.sh claude`
|
||||
- These scripts detect which AI agent is in use
|
||||
- Update the appropriate agent-specific context file
|
||||
- Add only new technology from current plan
|
||||
- Preserve manual additions between markers
|
||||
|
||||
**Output**: data-model.md, /contracts/*, quickstart.md, agent-specific file
|
||||
|
||||
## Key rules
|
||||
|
||||
- Use absolute paths
|
||||
- ERROR on gate failures or unresolved clarifications
|
||||
257
nextflow/.claude/commands/speckit.specify.md
Normal file
257
nextflow/.claude/commands/speckit.specify.md
Normal file
@@ -0,0 +1,257 @@
|
||||
---
|
||||
description: Create or update the feature specification from a natural language feature description.
|
||||
handoffs:
|
||||
- label: Build Technical Plan
|
||||
agent: speckit.plan
|
||||
prompt: Create a plan for the spec. I am building with...
|
||||
- label: Clarify Spec Requirements
|
||||
agent: speckit.clarify
|
||||
prompt: Clarify specification requirements
|
||||
send: true
|
||||
---
|
||||
|
||||
## User Input
|
||||
|
||||
```text
|
||||
$ARGUMENTS
|
||||
```
|
||||
|
||||
You **MUST** consider the user input before proceeding (if not empty).
|
||||
|
||||
## Outline
|
||||
|
||||
The text the user typed after `/speckit.specify` in the triggering message **is** the feature description. Assume you always have it available in this conversation even if `$ARGUMENTS` appears literally below. Do not ask the user to repeat it unless they provided an empty command.
|
||||
|
||||
Given that feature description, do this:
|
||||
|
||||
1. **Generate a concise short name** (2-4 words) for the branch:
|
||||
- Analyze the feature description and extract the most meaningful keywords
|
||||
- Create a 2-4 word short name that captures the essence of the feature
|
||||
- Use action-noun format when possible (e.g., "add-user-auth", "fix-payment-bug")
|
||||
- Preserve technical terms and acronyms (OAuth2, API, JWT, etc.)
|
||||
- Keep it concise but descriptive enough to understand the feature at a glance
|
||||
- Examples:
|
||||
- "I want to add user authentication" → "user-auth"
|
||||
- "Implement OAuth2 integration for the API" → "oauth2-api-integration"
|
||||
- "Create a dashboard for analytics" → "analytics-dashboard"
|
||||
- "Fix payment processing timeout bug" → "fix-payment-timeout"
|
||||
|
||||
2. **Check for existing branches before creating new one**:
|
||||
|
||||
a. First, fetch all remote branches to ensure we have the latest information:
|
||||
```bash
|
||||
git fetch --all --prune
|
||||
```
|
||||
|
||||
b. Find the highest feature number across all sources for the short-name:
|
||||
- Remote branches: `git ls-remote --heads origin | grep -E 'refs/heads/[0-9]+-<short-name>$'`
|
||||
- Local branches: `git branch | grep -E '^[* ]*[0-9]+-<short-name>$'`
|
||||
- Specs directories: Check for directories matching `specs/[0-9]+-<short-name>`
|
||||
|
||||
c. Determine the next available number:
|
||||
- Extract all numbers from all three sources
|
||||
- Find the highest number N
|
||||
- Use N+1 for the new branch number
|
||||
|
||||
d. Run the script `.specify/scripts/bash/create-new-feature.sh --json "$ARGUMENTS"` with the calculated number and short-name:
|
||||
- Pass `--number N+1` and `--short-name "your-short-name"` along with the feature description
|
||||
- Bash example: `.specify/scripts/bash/create-new-feature.sh --json "$ARGUMENTS" --json --number 5 --short-name "user-auth" "Add user authentication"`
|
||||
- PowerShell example: `.specify/scripts/bash/create-new-feature.sh --json "$ARGUMENTS" -Json -Number 5 -ShortName "user-auth" "Add user authentication"`
|
||||
|
||||
**IMPORTANT**:
|
||||
- Check all three sources (remote branches, local branches, specs directories) to find the highest number
|
||||
- Only match branches/directories with the exact short-name pattern
|
||||
- If no existing branches/directories found with this short-name, start with number 1
|
||||
- You must only ever run this script once per feature
|
||||
- The JSON is provided in the terminal as output - always refer to it to get the actual content you're looking for
|
||||
- The JSON output will contain BRANCH_NAME and SPEC_FILE paths
|
||||
- For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot")
|
||||
|
||||
3. Load `.specify/templates/spec-template.md` to understand required sections.
|
||||
|
||||
4. Follow this execution flow:
|
||||
|
||||
1. Parse user description from Input
|
||||
If empty: ERROR "No feature description provided"
|
||||
2. Extract key concepts from description
|
||||
Identify: actors, actions, data, constraints
|
||||
3. For unclear aspects:
|
||||
- Make informed guesses based on context and industry standards
|
||||
- Only mark with [NEEDS CLARIFICATION: specific question] if:
|
||||
- The choice significantly impacts feature scope or user experience
|
||||
- Multiple reasonable interpretations exist with different implications
|
||||
- No reasonable default exists
|
||||
- **LIMIT: Maximum 3 [NEEDS CLARIFICATION] markers total**
|
||||
- Prioritize clarifications by impact: scope > security/privacy > user experience > technical details
|
||||
4. Fill User Scenarios & Testing section
|
||||
If no clear user flow: ERROR "Cannot determine user scenarios"
|
||||
5. Generate Functional Requirements
|
||||
Each requirement must be testable
|
||||
Use reasonable defaults for unspecified details (document assumptions in Assumptions section)
|
||||
6. Define Success Criteria
|
||||
Create measurable, technology-agnostic outcomes
|
||||
Include both quantitative metrics (time, performance, volume) and qualitative measures (user satisfaction, task completion)
|
||||
Each criterion must be verifiable without implementation details
|
||||
7. Identify Key Entities (if data involved)
|
||||
8. Return: SUCCESS (spec ready for planning)
|
||||
|
||||
5. Write the specification to SPEC_FILE using the template structure, replacing placeholders with concrete details derived from the feature description (arguments) while preserving section order and headings.
|
||||
|
||||
6. **Specification Quality Validation**: After writing the initial spec, validate it against quality criteria:
|
||||
|
||||
a. **Create Spec Quality Checklist**: Generate a checklist file at `FEATURE_DIR/checklists/requirements.md` using the checklist template structure with these validation items:
|
||||
|
||||
```markdown
|
||||
# Specification Quality Checklist: [FEATURE NAME]
|
||||
|
||||
**Purpose**: Validate specification completeness and quality before proceeding to planning
|
||||
**Created**: [DATE]
|
||||
**Feature**: [Link to spec.md]
|
||||
|
||||
## Content Quality
|
||||
|
||||
- [ ] No implementation details (languages, frameworks, APIs)
|
||||
- [ ] Focused on user value and business needs
|
||||
- [ ] Written for non-technical stakeholders
|
||||
- [ ] All mandatory sections completed
|
||||
|
||||
## Requirement Completeness
|
||||
|
||||
- [ ] No [NEEDS CLARIFICATION] markers remain
|
||||
- [ ] Requirements are testable and unambiguous
|
||||
- [ ] Success criteria are measurable
|
||||
- [ ] Success criteria are technology-agnostic (no implementation details)
|
||||
- [ ] All acceptance scenarios are defined
|
||||
- [ ] Edge cases are identified
|
||||
- [ ] Scope is clearly bounded
|
||||
- [ ] Dependencies and assumptions identified
|
||||
|
||||
## Feature Readiness
|
||||
|
||||
- [ ] All functional requirements have clear acceptance criteria
|
||||
- [ ] User scenarios cover primary flows
|
||||
- [ ] Feature meets measurable outcomes defined in Success Criteria
|
||||
- [ ] No implementation details leak into specification
|
||||
|
||||
## Notes
|
||||
|
||||
- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan`
|
||||
```
|
||||
|
||||
b. **Run Validation Check**: Review the spec against each checklist item:
|
||||
- For each item, determine if it passes or fails
|
||||
- Document specific issues found (quote relevant spec sections)
|
||||
|
||||
c. **Handle Validation Results**:
|
||||
|
||||
- **If all items pass**: Mark checklist complete and proceed to step 6
|
||||
|
||||
- **If items fail (excluding [NEEDS CLARIFICATION])**:
|
||||
1. List the failing items and specific issues
|
||||
2. Update the spec to address each issue
|
||||
3. Re-run validation until all items pass (max 3 iterations)
|
||||
4. If still failing after 3 iterations, document remaining issues in checklist notes and warn user
|
||||
|
||||
- **If [NEEDS CLARIFICATION] markers remain**:
|
||||
1. Extract all [NEEDS CLARIFICATION: ...] markers from the spec
|
||||
2. **LIMIT CHECK**: If more than 3 markers exist, keep only the 3 most critical (by scope/security/UX impact) and make informed guesses for the rest
|
||||
3. For each clarification needed (max 3), present options to user in this format:
|
||||
|
||||
```markdown
|
||||
## Question [N]: [Topic]
|
||||
|
||||
**Context**: [Quote relevant spec section]
|
||||
|
||||
**What we need to know**: [Specific question from NEEDS CLARIFICATION marker]
|
||||
|
||||
**Suggested Answers**:
|
||||
|
||||
| Option | Answer | Implications |
|
||||
|--------|--------|--------------|
|
||||
| A | [First suggested answer] | [What this means for the feature] |
|
||||
| B | [Second suggested answer] | [What this means for the feature] |
|
||||
| C | [Third suggested answer] | [What this means for the feature] |
|
||||
| Custom | Provide your own answer | [Explain how to provide custom input] |
|
||||
|
||||
**Your choice**: _[Wait for user response]_
|
||||
```
|
||||
|
||||
4. **CRITICAL - Table Formatting**: Ensure markdown tables are properly formatted:
|
||||
- Use consistent spacing with pipes aligned
|
||||
- Each cell should have spaces around content: `| Content |` not `|Content|`
|
||||
- Header separator must have at least 3 dashes: `|--------|`
|
||||
- Test that the table renders correctly in markdown preview
|
||||
5. Number questions sequentially (Q1, Q2, Q3 - max 3 total)
|
||||
6. Present all questions together before waiting for responses
|
||||
7. Wait for user to respond with their choices for all questions (e.g., "Q1: A, Q2: Custom - [details], Q3: B")
|
||||
8. Update the spec by replacing each [NEEDS CLARIFICATION] marker with the user's selected or provided answer
|
||||
9. Re-run validation after all clarifications are resolved
|
||||
|
||||
d. **Update Checklist**: After each validation iteration, update the checklist file with current pass/fail status
|
||||
|
||||
7. Report completion with branch name, spec file path, checklist results, and readiness for the next phase (`/speckit.clarify` or `/speckit.plan`).
|
||||
|
||||
**NOTE:** The script creates and checks out the new branch and initializes the spec file before writing.
|
||||
|
||||
## General Guidelines
|
||||
|
||||
## Quick Guidelines
|
||||
|
||||
- Focus on **WHAT** users need and **WHY**.
|
||||
- Avoid HOW to implement (no tech stack, APIs, code structure).
|
||||
- Written for business stakeholders, not developers.
|
||||
- DO NOT create any checklists that are embedded in the spec. That will be a separate command.
|
||||
|
||||
### Section Requirements
|
||||
|
||||
- **Mandatory sections**: Must be completed for every feature
|
||||
- **Optional sections**: Include only when relevant to the feature
|
||||
- When a section doesn't apply, remove it entirely (don't leave as "N/A")
|
||||
|
||||
### For AI Generation
|
||||
|
||||
When creating this spec from a user prompt:
|
||||
|
||||
1. **Make informed guesses**: Use context, industry standards, and common patterns to fill gaps
|
||||
2. **Document assumptions**: Record reasonable defaults in the Assumptions section
|
||||
3. **Limit clarifications**: Maximum 3 [NEEDS CLARIFICATION] markers - use only for critical decisions that:
|
||||
- Significantly impact feature scope or user experience
|
||||
- Have multiple reasonable interpretations with different implications
|
||||
- Lack any reasonable default
|
||||
4. **Prioritize clarifications**: scope > security/privacy > user experience > technical details
|
||||
5. **Think like a tester**: Every vague requirement should fail the "testable and unambiguous" checklist item
|
||||
6. **Common areas needing clarification** (only if no reasonable default exists):
|
||||
- Feature scope and boundaries (include/exclude specific use cases)
|
||||
- User types and permissions (if multiple conflicting interpretations possible)
|
||||
- Security/compliance requirements (when legally/financially significant)
|
||||
|
||||
**Examples of reasonable defaults** (don't ask about these):
|
||||
|
||||
- Data retention: Industry-standard practices for the domain
|
||||
- Performance targets: Standard web/mobile app expectations unless specified
|
||||
- Error handling: User-friendly messages with appropriate fallbacks
|
||||
- Authentication method: Standard session-based or OAuth2 for web apps
|
||||
- Integration patterns: RESTful APIs unless specified otherwise
|
||||
|
||||
### Success Criteria Guidelines
|
||||
|
||||
Success criteria must be:
|
||||
|
||||
1. **Measurable**: Include specific metrics (time, percentage, count, rate)
|
||||
2. **Technology-agnostic**: No mention of frameworks, languages, databases, or tools
|
||||
3. **User-focused**: Describe outcomes from user/business perspective, not system internals
|
||||
4. **Verifiable**: Can be tested/validated without knowing implementation details
|
||||
|
||||
**Good examples**:
|
||||
|
||||
- "Users can complete checkout in under 3 minutes"
|
||||
- "System supports 10,000 concurrent users"
|
||||
- "95% of searches return results in under 1 second"
|
||||
- "Task completion rate improves by 40%"
|
||||
|
||||
**Bad examples** (implementation-focused):
|
||||
|
||||
- "API response time is under 200ms" (too technical, use "Users see results instantly")
|
||||
- "Database can handle 1000 TPS" (implementation detail, use user-facing metric)
|
||||
- "React components render efficiently" (framework-specific)
|
||||
- "Redis cache hit rate above 80%" (technology-specific)
|
||||
137
nextflow/.claude/commands/speckit.tasks.md
Normal file
137
nextflow/.claude/commands/speckit.tasks.md
Normal file
@@ -0,0 +1,137 @@
|
||||
---
|
||||
description: Generate an actionable, dependency-ordered tasks.md for the feature based on available design artifacts.
|
||||
handoffs:
|
||||
- label: Analyze For Consistency
|
||||
agent: speckit.analyze
|
||||
prompt: Run a project analysis for consistency
|
||||
send: true
|
||||
- label: Implement Project
|
||||
agent: speckit.implement
|
||||
prompt: Start the implementation in phases
|
||||
send: true
|
||||
---
|
||||
|
||||
## User Input
|
||||
|
||||
```text
|
||||
$ARGUMENTS
|
||||
```
|
||||
|
||||
You **MUST** consider the user input before proceeding (if not empty).
|
||||
|
||||
## Outline
|
||||
|
||||
1. **Setup**: Run `.specify/scripts/bash/check-prerequisites.sh --json` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
||||
|
||||
2. **Load design documents**: Read from FEATURE_DIR:
|
||||
- **Required**: plan.md (tech stack, libraries, structure), spec.md (user stories with priorities)
|
||||
- **Optional**: data-model.md (entities), contracts/ (API endpoints), research.md (decisions), quickstart.md (test scenarios)
|
||||
- Note: Not all projects have all documents. Generate tasks based on what's available.
|
||||
|
||||
3. **Execute task generation workflow**:
|
||||
- Load plan.md and extract tech stack, libraries, project structure
|
||||
- Load spec.md and extract user stories with their priorities (P1, P2, P3, etc.)
|
||||
- If data-model.md exists: Extract entities and map to user stories
|
||||
- If contracts/ exists: Map endpoints to user stories
|
||||
- If research.md exists: Extract decisions for setup tasks
|
||||
- Generate tasks organized by user story (see Task Generation Rules below)
|
||||
- Generate dependency graph showing user story completion order
|
||||
- Create parallel execution examples per user story
|
||||
- Validate task completeness (each user story has all needed tasks, independently testable)
|
||||
|
||||
4. **Generate tasks.md**: Use `.specify.specify/templates/tasks-template.md` as structure, fill with:
|
||||
- Correct feature name from plan.md
|
||||
- Phase 1: Setup tasks (project initialization)
|
||||
- Phase 2: Foundational tasks (blocking prerequisites for all user stories)
|
||||
- Phase 3+: One phase per user story (in priority order from spec.md)
|
||||
- Each phase includes: story goal, independent test criteria, tests (if requested), implementation tasks
|
||||
- Final Phase: Polish & cross-cutting concerns
|
||||
- All tasks must follow the strict checklist format (see Task Generation Rules below)
|
||||
- Clear file paths for each task
|
||||
- Dependencies section showing story completion order
|
||||
- Parallel execution examples per story
|
||||
- Implementation strategy section (MVP first, incremental delivery)
|
||||
|
||||
5. **Report**: Output path to generated tasks.md and summary:
|
||||
- Total task count
|
||||
- Task count per user story
|
||||
- Parallel opportunities identified
|
||||
- Independent test criteria for each story
|
||||
- Suggested MVP scope (typically just User Story 1)
|
||||
- Format validation: Confirm ALL tasks follow the checklist format (checkbox, ID, labels, file paths)
|
||||
|
||||
Context for task generation: $ARGUMENTS
|
||||
|
||||
The tasks.md should be immediately executable - each task must be specific enough that an LLM can complete it without additional context.
|
||||
|
||||
## Task Generation Rules
|
||||
|
||||
**CRITICAL**: Tasks MUST be organized by user story to enable independent implementation and testing.
|
||||
|
||||
**Tests are OPTIONAL**: Only generate test tasks if explicitly requested in the feature specification or if user requests TDD approach.
|
||||
|
||||
### Checklist Format (REQUIRED)
|
||||
|
||||
Every task MUST strictly follow this format:
|
||||
|
||||
```text
|
||||
- [ ] [TaskID] [P?] [Story?] Description with file path
|
||||
```
|
||||
|
||||
**Format Components**:
|
||||
|
||||
1. **Checkbox**: ALWAYS start with `- [ ]` (markdown checkbox)
|
||||
2. **Task ID**: Sequential number (T001, T002, T003...) in execution order
|
||||
3. **[P] marker**: Include ONLY if task is parallelizable (different files, no dependencies on incomplete tasks)
|
||||
4. **[Story] label**: REQUIRED for user story phase tasks only
|
||||
- Format: [US1], [US2], [US3], etc. (maps to user stories from spec.md)
|
||||
- Setup phase: NO story label
|
||||
- Foundational phase: NO story label
|
||||
- User Story phases: MUST have story label
|
||||
- Polish phase: NO story label
|
||||
5. **Description**: Clear action with exact file path
|
||||
|
||||
**Examples**:
|
||||
|
||||
- ✅ CORRECT: `- [ ] T001 Create project structure per implementation plan`
|
||||
- ✅ CORRECT: `- [ ] T005 [P] Implement authentication middleware in src/middleware/auth.py`
|
||||
- ✅ CORRECT: `- [ ] T012 [P] [US1] Create User model in src/models/user.py`
|
||||
- ✅ CORRECT: `- [ ] T014 [US1] Implement UserService in src/services/user_service.py`
|
||||
- ❌ WRONG: `- [ ] Create User model` (missing ID and Story label)
|
||||
- ❌ WRONG: `T001 [US1] Create model` (missing checkbox)
|
||||
- ❌ WRONG: `- [ ] [US1] Create User model` (missing Task ID)
|
||||
- ❌ WRONG: `- [ ] T001 [US1] Create model` (missing file path)
|
||||
|
||||
### Task Organization
|
||||
|
||||
1. **From User Stories (spec.md)** - PRIMARY ORGANIZATION:
|
||||
- Each user story (P1, P2, P3...) gets its own phase
|
||||
- Map all related components to their story:
|
||||
- Models needed for that story
|
||||
- Services needed for that story
|
||||
- Endpoints/UI needed for that story
|
||||
- If tests requested: Tests specific to that story
|
||||
- Mark story dependencies (most stories should be independent)
|
||||
|
||||
2. **From Contracts**:
|
||||
- Map each contract/endpoint → to the user story it serves
|
||||
- If tests requested: Each contract → contract test task [P] before implementation in that story's phase
|
||||
|
||||
3. **From Data Model**:
|
||||
- Map each entity to the user story(ies) that need it
|
||||
- If entity serves multiple stories: Put in earliest story or Setup phase
|
||||
- Relationships → service layer tasks in appropriate story phase
|
||||
|
||||
4. **From Setup/Infrastructure**:
|
||||
- Shared infrastructure → Setup phase (Phase 1)
|
||||
- Foundational/blocking tasks → Foundational phase (Phase 2)
|
||||
- Story-specific setup → within that story's phase
|
||||
|
||||
### Phase Structure
|
||||
|
||||
- **Phase 1**: Setup (project initialization)
|
||||
- **Phase 2**: Foundational (blocking prerequisites - MUST complete before user stories)
|
||||
- **Phase 3+**: User Stories in priority order (P1, P2, P3...)
|
||||
- Within each story: Tests (if requested) → Models → Services → Endpoints → Integration
|
||||
- Each phase should be a complete, independently testable increment
|
||||
- **Final Phase**: Polish & Cross-Cutting Concerns
|
||||
28
nextflow/.claude/commands/speckit.taskstoissues.md
Normal file
28
nextflow/.claude/commands/speckit.taskstoissues.md
Normal file
@@ -0,0 +1,28 @@
|
||||
---
|
||||
description: Convert existing tasks into actionable, dependency-ordered GitHub issues for the feature based on available design artifacts.
|
||||
tools: ['github/github-mcp-server/issue_write']
|
||||
---
|
||||
|
||||
## User Input
|
||||
|
||||
```text
|
||||
$ARGUMENTS
|
||||
```
|
||||
|
||||
You **MUST** consider the user input before proceeding (if not empty).
|
||||
|
||||
## Outline
|
||||
|
||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
||||
1. From the executed script, extract the path to **tasks**.
|
||||
1. Get the Git remote by running:
|
||||
|
||||
```bash
|
||||
git config --get remote.origin.url
|
||||
```
|
||||
|
||||
**ONLY PROCEED TO NEXT STEPS IF THE REMOTE IS A GITHUB URL**
|
||||
|
||||
1. For each task in the list, use the GitHub MCP server to create a new issue in the repository that is representative of the Git remote.
|
||||
|
||||
**UNDER NO CIRCUMSTANCES EVER CREATE ISSUES IN REPOSITORIES THAT DO NOT MATCH THE REMOTE URL**
|
||||
6
nextflow/.codespellrc
Normal file
6
nextflow/.codespellrc
Normal file
@@ -0,0 +1,6 @@
|
||||
[codespell]
|
||||
skip = .git,*.pdf,*.svg,timeline-expected.html,*.fq,*.min.js,ScriptDslTest.groovy
|
||||
# some cases where we need to catch using regex
|
||||
ignore-regex = \bhel\*|fo\\
|
||||
# some variables, names, etc to ignore
|
||||
ignore-words-list = splitted,ois,tre,marge,smoot,afile,bams,bais,pre-pending,re-use
|
||||
8
nextflow/.editorconfig
Normal file
8
nextflow/.editorconfig
Normal file
@@ -0,0 +1,8 @@
|
||||
[*]
|
||||
charset = utf-8
|
||||
indent_size = 4
|
||||
indent_style = space
|
||||
tab_width = 4
|
||||
end_of_line = lf
|
||||
insert_final_newline = true
|
||||
trim_trailing_whitespace = true
|
||||
7
nextflow/.github/CODEOWNERS
vendored
Normal file
7
nextflow/.github/CODEOWNERS
vendored
Normal file
@@ -0,0 +1,7 @@
|
||||
# DOCUMENTATION -------------------------------------------------------
|
||||
# Docs folder ownership
|
||||
/docs/ @nextflow-io/docs
|
||||
|
||||
# COMPILER ------------------------------------------------------------
|
||||
# nf-lang module
|
||||
/modules/nf-lang/ @nextflow-io/lang
|
||||
31
nextflow/.github/ISSUE_TEMPLATE/bug_report.md
vendored
Normal file
31
nextflow/.github/ISSUE_TEMPLATE/bug_report.md
vendored
Normal file
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: Bug report
|
||||
about: Report a bug to help us improve
|
||||
---
|
||||
|
||||
## Bug report
|
||||
|
||||
(Please follow this template by replacing the text between parentheses with the requested information)
|
||||
|
||||
### Expected behavior and actual behavior
|
||||
|
||||
(Give a brief description of the expected behavior and actual behavior)
|
||||
|
||||
### Steps to reproduce the problem
|
||||
|
||||
(Provide a test case that reproduces the problem either with a self-contained script or GitHub repository)
|
||||
|
||||
### Program output
|
||||
|
||||
(Copy and paste the output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the `.nextflow.log` file.)
|
||||
|
||||
### Environment
|
||||
|
||||
* Nextflow version: [?]
|
||||
* Java version: [?]
|
||||
* Operating system: [macOS, Linux, etc]
|
||||
* Bash version: (use the command `$SHELL --version`)
|
||||
|
||||
### Additional context
|
||||
|
||||
(Add any other context about the problem here)
|
||||
11
nextflow/.github/ISSUE_TEMPLATE/config.yml
vendored
Normal file
11
nextflow/.github/ISSUE_TEMPLATE/config.yml
vendored
Normal file
@@ -0,0 +1,11 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: General question
|
||||
url: https://community.seqera.io/nextflow
|
||||
about: Ask for help with Nextflow language and usage
|
||||
- name: Nextflow official website
|
||||
url: https://nextflow.io/
|
||||
about: Documentation and tutorials
|
||||
- name: Nextflow training
|
||||
url: https://training.nextflow.io/
|
||||
about: Learn Nextflow with hands-on tutorials
|
||||
16
nextflow/.github/ISSUE_TEMPLATE/new_feature.md
vendored
Normal file
16
nextflow/.github/ISSUE_TEMPLATE/new_feature.md
vendored
Normal file
@@ -0,0 +1,16 @@
|
||||
---
|
||||
name: New feature
|
||||
about: Propose a new feature or enhancement
|
||||
---
|
||||
|
||||
## New feature
|
||||
|
||||
(Hi! Thanks for using Nextflow and for proposing a new feature or enhancement. Please replace this text with a brief description of your proposal.)
|
||||
|
||||
## Use case
|
||||
|
||||
(What's the main use case and deployment scenario addressed by this proposal)
|
||||
|
||||
## Suggested implementation
|
||||
|
||||
(Highlight the main building blocks of a possible implementation and/or related components)
|
||||
7
nextflow/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md
vendored
Normal file
7
nextflow/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md
vendored
Normal file
@@ -0,0 +1,7 @@
|
||||
Hi! Thanks for contributing to Nextflow.
|
||||
|
||||
When submitting a Pull Request, please sign-off the DCO [1] to certify that you are the author of the contribution and you adhere to Nextflow's open source license [2] by adding a `Signed-off-by` line to the contribution commit message. See [3] for more details.
|
||||
|
||||
1. https://developercertificate.org/
|
||||
2. https://github.com/nextflow-io/nextflow/blob/master/COPYING
|
||||
3. https://github.com/apps/dco
|
||||
15
nextflow/.github/workflows/README.md
vendored
Normal file
15
nextflow/.github/workflows/README.md
vendored
Normal file
@@ -0,0 +1,15 @@
|
||||
# Action
|
||||
|
||||
## Syntax
|
||||
|
||||
https://help.github.com/en/articles/workflow-syntax-for-github-actions
|
||||
https://help.github.com/en/articles/contexts-and-expression-syntax-for-github-actions
|
||||
https://help.github.com/en/articles/virtual-environments-for-github-actions#environment-variables
|
||||
https://help.github.com/en/articles/configuring-docker-for-use-with-github-package-registry
|
||||
https://help.github.com/en/articles/virtual-environments-for-github-actions#creating-and-using-secrets-encrypted-variables
|
||||
|
||||
## Java
|
||||
|
||||
Java VMs has to match the ones at this link https://static.azul.com/zulu/bin
|
||||
|
||||
Check the name *-jdk(x.y.z)
|
||||
296
nextflow/.github/workflows/build.yml
vendored
Normal file
296
nextflow/.github/workflows/build.yml
vendored
Normal file
@@ -0,0 +1,296 @@
|
||||
name: Nextflow CI
|
||||
# read more here: https://help.github.com/en/articles/workflow-syntax-for-github-actions#on
|
||||
|
||||
# Note: We don't use the `on: path` option for docs,
|
||||
# because the Build steps are *required* tests.
|
||||
# Instead, we trigger + skip the tests if the only changes
|
||||
# are in the docs folder. GitHub treats this as passing.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'test*'
|
||||
- 'dev*'
|
||||
- 'STABLE-*'
|
||||
pull_request:
|
||||
types: [opened, reopened, synchronize]
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
build:
|
||||
name: Build
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 100
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
java_version: [17, 25]
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 1
|
||||
submodules: true
|
||||
|
||||
- name: Get the commit message
|
||||
id: get_commit_message
|
||||
run: |
|
||||
if [ "${{ github.event_name }}" = "pull_request" ]; then
|
||||
echo "GitHub event=pull_request"
|
||||
COMMIT_SHA="${{ github.event.pull_request.head.sha }}"
|
||||
COMMIT_MESSAGE="$(curl -s \
|
||||
-H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
|
||||
https://api.github.com/repos/${{ github.repository }}/commits/$COMMIT_SHA | jq -r '.commit.message' | head -n 1)"
|
||||
echo "Commit message=$(printf "%s" "$COMMIT_MESSAGE")"
|
||||
echo "commit_message=$(printf "%s" "$COMMIT_MESSAGE")" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "GitHub event=${{ github.event_name }}"
|
||||
# Extract only the first line of the commit message
|
||||
COMMIT_MESSAGE="$(git log -1 --pretty=format:'%s')"
|
||||
echo "Commit message=$(printf "%s" "$COMMIT_MESSAGE")"
|
||||
echo "commit_message=$(printf "%s" "$COMMIT_MESSAGE")" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Setup env
|
||||
run: |
|
||||
rm -f $HOME/.gitconfig;
|
||||
mkdir -p "$HOME/.nextflow";
|
||||
echo "providers.github.auth='$NXF_GITHUB_ACCESS_TOKEN'" > "$HOME/.nextflow/scm"
|
||||
env:
|
||||
NXF_GITHUB_ACCESS_TOKEN: ${{ secrets.NXF_GITHUB_ACCESS_TOKEN }}
|
||||
|
||||
- name: Setup Java ${{ matrix.java_version }}
|
||||
uses: actions/setup-java@v4
|
||||
with:
|
||||
java-version: ${{matrix.java_version}}
|
||||
distribution: 'temurin'
|
||||
architecture: x64
|
||||
cache: gradle
|
||||
|
||||
- name: Compile
|
||||
run: make assemble
|
||||
|
||||
- name: Test
|
||||
run: |
|
||||
env | sort
|
||||
# configure test env
|
||||
if [[ "$GOOGLE_SECRET" ]]; then
|
||||
echo $GOOGLE_SECRET | base64 -d > $PWD/google_credentials.json
|
||||
export GOOGLE_APPLICATION_CREDENTIALS=$PWD/google_credentials.json
|
||||
fi
|
||||
# run tests
|
||||
make test
|
||||
env:
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
AWS_S3FS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
AWS_S3FS_SECRET_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
NXF_BITBUCKET_ACCESS_TOKEN: ${{ secrets.NXF_BITBUCKET_ACCESS_TOKEN }}
|
||||
NXF_GITHUB_ACCESS_TOKEN: ${{ secrets.NXF_GITHUB_ACCESS_TOKEN }}
|
||||
NXF_GITLAB_ACCESS_TOKEN: ${{ secrets.NXF_GITLAB_ACCESS_TOKEN }}
|
||||
NXF_AZURE_REPOS_TOKEN: ${{ secrets.NXF_AZURE_REPOS_TOKEN }}
|
||||
GOOGLE_SECRET: ${{ secrets.GOOGLE_SECRET }}
|
||||
AZURE_STORAGE_ACCOUNT_NAME: nfazurestore
|
||||
AZURE_STORAGE_ACCOUNT_KEY: ${{ secrets.AZURE_STORAGE_ACCOUNT_KEY }}
|
||||
AZURE_BATCH_ACCOUNT_NAME: nfbatchtest
|
||||
AZURE_BATCH_ACCOUNT_KEY: ${{ secrets.AZURE_BATCH_ACCOUNT_KEY }}
|
||||
|
||||
- name: Publish tests report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: report-unit-tests-jdk-${{ matrix.java_version }}
|
||||
path: |
|
||||
**/build/reports/tests/test
|
||||
|
||||
outputs:
|
||||
commit_message: ${{ steps.get_commit_message.outputs.commit_message }}
|
||||
|
||||
test:
|
||||
if: ${{ !contains(needs.build.outputs.commit_message, '[ci fast]') }}
|
||||
needs: build
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
java_version: [17, 25]
|
||||
test_mode: ["test_integration", "test_parser_v2", "test_docs", "test_aws", "test_azure", "test_google", "test_wave"]
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 1
|
||||
submodules: true
|
||||
|
||||
- name: Setup env
|
||||
run: |
|
||||
rm -f $HOME/.gitconfig;
|
||||
mkdir -p "$HOME/.nextflow";
|
||||
echo "providers.github.auth='$NXF_GITHUB_ACCESS_TOKEN'" > "$HOME/.nextflow/scm"
|
||||
env:
|
||||
NXF_GITHUB_ACCESS_TOKEN: ${{ secrets.NXF_GITHUB_ACCESS_TOKEN }}
|
||||
|
||||
- name: Setup Java ${{ matrix.java_version }}
|
||||
uses: actions/setup-java@v4
|
||||
with:
|
||||
java-version: ${{matrix.java_version}}
|
||||
distribution: 'temurin'
|
||||
architecture: x64
|
||||
cache: gradle
|
||||
|
||||
- name: Run tests
|
||||
run: |
|
||||
env | sort
|
||||
# configure test env
|
||||
if [[ "$GOOGLE_SECRET" ]]; then
|
||||
echo $GOOGLE_SECRET | base64 -d > $PWD/google_credentials.json
|
||||
export GOOGLE_APPLICATION_CREDENTIALS=$PWD/google_credentials.json
|
||||
fi
|
||||
cat $HOME/.nextflow/scm
|
||||
make clean assemble install
|
||||
bash test-ci.sh
|
||||
env:
|
||||
TEST_JDK: ${{ matrix.java_version }}
|
||||
TEST_MODE: ${{ matrix.test_mode }}
|
||||
GRADLE_OPTS: '-Dorg.gradle.daemon=false'
|
||||
TOWER_ACCESS_TOKEN: ${{ secrets.TOWER_ACCESS_TOKEN }}
|
||||
AWS_DEFAULT_REGION: eu-west-1
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
NXF_BITBUCKET_ACCESS_TOKEN: ${{ secrets.NXF_BITBUCKET_ACCESS_TOKEN }}
|
||||
NXF_GITHUB_ACCESS_TOKEN: ${{ secrets.NXF_GITHUB_ACCESS_TOKEN }}
|
||||
NXF_GITLAB_ACCESS_TOKEN: ${{ secrets.NXF_GITLAB_ACCESS_TOKEN }}
|
||||
NXF_AZURE_REPOS_TOKEN: ${{ secrets.NXF_AZURE_REPOS_TOKEN }}
|
||||
GOOGLE_SECRET: ${{ secrets.GOOGLE_SECRET }}
|
||||
AZURE_STORAGE_ACCOUNT_NAME: nfazurestore
|
||||
AZURE_STORAGE_ACCOUNT_KEY: ${{ secrets.AZURE_STORAGE_ACCOUNT_KEY }}
|
||||
AZURE_BATCH_ACCOUNT_NAME: nfbatchtest
|
||||
AZURE_BATCH_ACCOUNT_KEY: ${{ secrets.AZURE_BATCH_ACCOUNT_KEY }}
|
||||
|
||||
- name: Tar integration tests
|
||||
if: always()
|
||||
run: |
|
||||
tar -cvf integration-tests.tar.gz tests/checks
|
||||
tar -cvf validation-tests.tar.gz validation
|
||||
|
||||
- name: Publish tests report
|
||||
uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: report-${{ matrix.test_mode }}-jdk-${{ matrix.java_version }}
|
||||
path: |
|
||||
validation-tests.tar.gz
|
||||
integration-tests.tar.gz
|
||||
|
||||
test-e2e:
|
||||
if: ${{ contains(needs.build.outputs.commit_message,'[e2e stage]') || contains(needs.build.outputs.commit_message,'[e2e prod]') }}
|
||||
needs: build
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
actions: write # Allow writing to actions
|
||||
contents: write # Allow writing to repository contents
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 1
|
||||
submodules: true
|
||||
|
||||
- name: Setup Java 17
|
||||
uses: actions/setup-java@v4
|
||||
with:
|
||||
java-version: 17
|
||||
distribution: 'temurin'
|
||||
architecture: x64
|
||||
cache: gradle
|
||||
|
||||
- name: Setup env
|
||||
run: |
|
||||
wget -q -O wave https://github.com/seqeralabs/wave-cli/releases/download/v1.4.1/wave-1.4.1-linux-x86_64
|
||||
chmod +x wave
|
||||
mv wave /usr/local/bin/
|
||||
echo "COMMIT_MESSAGE=\"${{ needs.build.outputs.commit_message }}\"" >> $GITHUB_ENV
|
||||
|
||||
- name : Docker Login to Seqera public CR
|
||||
uses : docker/login-action@v3
|
||||
with :
|
||||
registry : "public.cr.seqera.io"
|
||||
username : "public-cr-admin"
|
||||
password : ${{ secrets.SEQERA_PUBLIC_CR_PASSWORD }}
|
||||
|
||||
- name: Launch tests
|
||||
run: |
|
||||
cd test-e2e
|
||||
bash run.sh
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.AUTOMATION_GITHUB_TOKEN }}
|
||||
GRADLE_OPTS: '-Dorg.gradle.daemon=false'
|
||||
|
||||
release:
|
||||
if: ${{ always() && contains(needs.build.outputs.commit_message, '[release]') && needs.build.result == 'success' && (needs.test.result == 'success' || needs.test.result == 'skipped') }}
|
||||
needs: [build, test]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
actions: write
|
||||
contents: write
|
||||
packages: write
|
||||
pull-requests: write
|
||||
issues: write
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
submodules: true
|
||||
|
||||
- name: Setup Java 17
|
||||
uses: actions/setup-java@v4
|
||||
with:
|
||||
java-version: 17
|
||||
distribution: 'temurin'
|
||||
architecture: x64
|
||||
cache: gradle
|
||||
|
||||
- name: Configure Git
|
||||
run: |
|
||||
git config --global user.name "${{ github.event.pusher.name || github.actor }}"
|
||||
git config --global user.email "${{ github.event.pusher.email || format('{0}@users.noreply.github.com', github.actor) }}"
|
||||
|
||||
- name: Docker Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ vars.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Docker Login to Seqera public CR
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: "public.cr.seqera.io"
|
||||
username: ${{ vars.SEQERA_PUBLIC_CR_USERNAME }}
|
||||
password: ${{ secrets.SEQERA_PUBLIC_CR_PASSWORD }}
|
||||
|
||||
- name: Run release
|
||||
run: |
|
||||
echo "Starting release process..."
|
||||
echo "npr.apiUrl=$NPR_API_URL" >> gradle.properties
|
||||
echo "npr.apiKey=$NPR_API_KEY" >> gradle.properties
|
||||
bash release.sh
|
||||
env:
|
||||
GRADLE_OPTS: '-Dorg.gradle.daemon=false'
|
||||
AWS_JAVA_V1_DISABLE_DEPRECATION_ANNOUNCEMENT: 'true'
|
||||
# credentials to pubslish nextflow assets
|
||||
NXF_AWS_ACCESS: ${{ vars.NXF_AWS_ACCESS }}
|
||||
NXF_AWS_SECRET: ${{ secrets.NXF_AWS_SECRET }}
|
||||
# credentials to publish maven libraries
|
||||
AWS_ACCESS_KEY_ID: ${{ vars.SEQERA_MAVEN_ACCESS_KEY }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.SEQERA_MAVEN_SECRET_KEY }}
|
||||
# plugin registry
|
||||
NPR_API_URL: ${{ vars.NPR_API_URL }}
|
||||
NPR_API_KEY: ${{ secrets.NPR_API_KEY }}
|
||||
# GitHub secrets
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
19
nextflow/.github/workflows/cffconvert.yml
vendored
Normal file
19
nextflow/.github/workflows/cffconvert.yml
vendored
Normal file
@@ -0,0 +1,19 @@
|
||||
name: cffconvert
|
||||
|
||||
on:
|
||||
push:
|
||||
paths:
|
||||
- CITATION.cff
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
name: "validate"
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Check out a copy of the repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Check whether the citation metadata from CITATION.cff is valid
|
||||
uses: citation-file-format/cffconvert-github-action@2.0.0
|
||||
with:
|
||||
args: "--validate"
|
||||
36
nextflow/.github/workflows/claude.yml
vendored
Normal file
36
nextflow/.github/workflows/claude.yml
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
name: Claude PR Assistant
|
||||
|
||||
on:
|
||||
issue_comment:
|
||||
types: [created]
|
||||
pull_request_review_comment:
|
||||
types: [created]
|
||||
issues:
|
||||
types: [opened, assigned]
|
||||
pull_request_review:
|
||||
types: [submitted]
|
||||
|
||||
jobs:
|
||||
claude-code-action:
|
||||
if: |
|
||||
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
|
||||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
|
||||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
|
||||
(github.event_name == 'issues' && contains(github.event.issue.body, '@claude'))
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
issues: read
|
||||
id-token: write
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Run Claude PR Action
|
||||
uses: anthropics/claude-code-action@beta
|
||||
with:
|
||||
anthropic_api_key: ${{ secrets.NEXTFLOW_ANTHROPIC_API_KEY }}
|
||||
timeout_minutes: "60"
|
||||
22
nextflow/.github/workflows/docs.yml
vendored
Normal file
22
nextflow/.github/workflows/docs.yml
vendored
Normal file
@@ -0,0 +1,22 @@
|
||||
name: Docs CI
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, reopened, synchronize]
|
||||
paths:
|
||||
- 'docs/**'
|
||||
workflow_dispatch:
|
||||
jobs:
|
||||
docs-build:
|
||||
name: Build
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.9'
|
||||
|
||||
- name: Test docs build
|
||||
run: |
|
||||
cd docs/
|
||||
pip install -r requirements.txt
|
||||
make clean html
|
||||
21
nextflow/.github/workflows/stale.yml
vendored
Normal file
21
nextflow/.github/workflows/stale.yml
vendored
Normal file
@@ -0,0 +1,21 @@
|
||||
name: 'Mark stale issues and PRs'
|
||||
on:
|
||||
schedule:
|
||||
- cron: '30 1 * * *'
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
stale:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/stale@v9
|
||||
with:
|
||||
days-before-stale: 180
|
||||
days-before-close: -1
|
||||
stale-issue-label: stale
|
||||
stale-issue-message: ''
|
||||
stale-pr-label: stale
|
||||
stale-pr-message: ''
|
||||
exempt-issue-labels: bug,planned,security
|
||||
exempt-all-milestones: true
|
||||
exempt-all-assignees: true
|
||||
48
nextflow/.gitignore
vendored
Normal file
48
nextflow/.gitignore
vendored
Normal file
@@ -0,0 +1,48 @@
|
||||
.idea/**
|
||||
.gradle
|
||||
.*.log*
|
||||
.cache/*
|
||||
.vagrant/*
|
||||
.cache/*
|
||||
.nextflow*
|
||||
.node-nextflow*
|
||||
.devcontainer
|
||||
.vscode/*
|
||||
.lineage/
|
||||
*/*/bin/*
|
||||
**/build/**
|
||||
build/**
|
||||
modules/**/build/
|
||||
modules/**/*.iml
|
||||
modules/nextflow/work/
|
||||
**/out/*
|
||||
tmp/*
|
||||
work/*
|
||||
nextflow.eml
|
||||
nextflow.iml
|
||||
Vagrantfile
|
||||
resources/*
|
||||
test/*
|
||||
tools/*
|
||||
tests_other
|
||||
other
|
||||
docs/_build/
|
||||
./*.txt
|
||||
errors/
|
||||
misc/
|
||||
cloud/my-*
|
||||
docker-test
|
||||
docker/dist
|
||||
docker/nextflow
|
||||
temp
|
||||
.dockerignore
|
||||
.launch.classpath
|
||||
plugins-prod
|
||||
/minio
|
||||
/sandbox
|
||||
/wave-tests
|
||||
/test-sched
|
||||
/test-module
|
||||
/results
|
||||
/x/*
|
||||
mise.toml
|
||||
198
nextflow/.specify/memory/constitution.md
Normal file
198
nextflow/.specify/memory/constitution.md
Normal file
@@ -0,0 +1,198 @@
|
||||
# Nextflow Development Constitution
|
||||
|
||||
<!--
|
||||
SYNC IMPACT REPORT
|
||||
==================
|
||||
Version Change: INITIAL → 1.0.0 (Initial constitution)
|
||||
Modified Principles: N/A (new constitution)
|
||||
Added Sections:
|
||||
- All principles (I-VII)
|
||||
- Development Workflow
|
||||
- Quality Standards
|
||||
- Governance
|
||||
|
||||
Removed Sections: N/A (initial version)
|
||||
|
||||
Templates Status:
|
||||
✅ plan-template.md - Reviewed, aligned with modular architecture and testing principles
|
||||
✅ spec-template.md - Reviewed, aligned with user scenario focus and requirements structure
|
||||
✅ tasks-template.md - Reviewed, aligned with test-driven and parallel development principles
|
||||
✅ agent-file-template.md - Reviewed, no agent-specific conflicts
|
||||
✅ checklist-template.md - Reviewed, compatible with quality standards
|
||||
|
||||
Follow-up TODOs:
|
||||
- None at this time
|
||||
==================
|
||||
-->
|
||||
|
||||
## Core Principles
|
||||
|
||||
### I. Modular Architecture
|
||||
|
||||
Nextflow MUST maintain a clear separation between core functionality and extensions through its modular architecture:
|
||||
|
||||
- **Core modules** (`modules/`) contain essential functionality: core workflow runtime & plugin system (nextflow), shared utilities (nf-commons), language parsing (nf-lang), HTTP filesystem support (nf-httpfs), and lineage tracking (nf-lineage)
|
||||
- **Core plugins** (`plugins/`) provides cloud provider integrations (AWS, Azure, GCP), execution platforms (Kubernetes), and specialized services (Seqera Platform, Wave container management)
|
||||
- New features MUST be evaluated for placement: core features belong in `modules/`, specialized/cloud-specific features belong in `plugins/`
|
||||
- Each module and plugin MUST be independently buildable and testable
|
||||
- Plugin dependencies MUST be explicitly declared in `build.gradle` with semantic versioning
|
||||
|
||||
**Rationale**: This architecture enables independent development of cloud provider features without core engine changes, supports third-party plugin development, and maintains a clean separation of concerns across a large multi-module codebase.
|
||||
|
||||
### II. Test-Driven Quality Assurance (NON-NEGOTIABLE)
|
||||
|
||||
Testing MUST be comprehensive and multi-layered before any code is merged:
|
||||
|
||||
- **Unit tests** MUST use Spock Framework for all Groovy code, be independently executable, and achieve meaningful coverage (measured via JaCoCo)
|
||||
- **Integration tests** (`tests/` directory) MUST validate end-to-end workflows using actual `.nf` scripts with expected outputs
|
||||
- **Smoke tests** (`make smoke` or `NXF_SMOKE=1`) MUST be available to skip long-running and cloud-dependent tests during rapid development
|
||||
- **Cloud validation tests** (`validation/` directory) MUST verify cloud provider integrations end-to-end before release
|
||||
- **Documentation tests** (`docs/snippets/`) MUST ensure all documentation examples remain functional
|
||||
- All tests MUST pass before commits, and `make test` MUST be run locally before pushing
|
||||
|
||||
**Rationale**: Scientific workflows demand reliability and reproducibility. Multi-layered testing catches issues at appropriate levels: unit tests for logic, integration tests for workflow correctness, and validation tests for cloud provider compatibility.
|
||||
|
||||
### III. Dataflow Programming Model
|
||||
|
||||
Nextflow's core abstraction MUST adhere to following principles and guidelines:
|
||||
|
||||
- The dataflow programming model is the fundamental abstraction in the Nextflow programming model.
|
||||
- Workflows are defined as dataflow graphs where data flows between processes.
|
||||
- Processes MUST be stateless, side-effect-free transformations that communicate via channels.
|
||||
- Workflows should be defined in a platform agnostic manner to enable portability across clusters and clouds.
|
||||
- The DSL MUST prioritize expressiveness for concurrent and parallel pipeline definition.
|
||||
- Changes to the language parser (ANTLR grammars in `nf-lang`) MUST preserve backward compatibility with existing pipelines unless explicitly versioned (DSL1 vs DSL2).
|
||||
- Concurrency primitives (GPars actors/dataflow) MUST be used correctly to maintain the dataflow semantics.
|
||||
|
||||
**Rationale**: The dataflow model is Nextflow's fundamental abstraction, enabling automatic parallelization and distribution. Preserving this model ensures existing scientific pipelines continue to work and users can reason about workflow behavior.
|
||||
|
||||
### IV. Apache 2.0 License Compliance
|
||||
|
||||
All source code MUST include Apache 2.0 license headers:
|
||||
|
||||
- Every source file MUST begin with the Apache 2.0 license header
|
||||
- All contributions MUST comply with Apache 2.0 terms
|
||||
- Third-party dependencies MUST use compatible licenses
|
||||
- License compliance MUST be verified during code review
|
||||
|
||||
**Rationale**: Legal clarity protects both contributors and users. Consistent licensing enables academic and commercial use, which is critical for scientific software adoption.
|
||||
|
||||
### V. Developer Certificate of Origin (DCO) Sign-off
|
||||
|
||||
All commits MUST be signed with DCO certification:
|
||||
|
||||
- Contributors MUST certify they have the right to submit the code by using `git commit -s` or `git commit --signoff`
|
||||
- Every commit message MUST include a `Signed-off-by` line
|
||||
- The DCO bot MUST verify sign-off before any PR can be merged
|
||||
- Contributors MUST NOT bypass the DCO requirement
|
||||
|
||||
**Rationale**: DCO provides legal protection and clear chain of custody for contributions, which is essential for open-source projects with diverse contributors.
|
||||
|
||||
### VI. Semantic Versioning and Release Discipline
|
||||
|
||||
Version management MUST follow strict semantic versioning with calendar-based releases:
|
||||
|
||||
- **Project versions** use calendar-based scheme: `YY.MM.PATCH` where April (`.04.`) and October (`.10.`) are stable releases, all other months use `-edge` suffix (e.g., `25.09.0-edge`)
|
||||
- **Plugin versions** MUST use semantic versioning (`MAJOR.MINOR.PATCH`)
|
||||
- Version changes MUST be documented in `changelog.txt` files (both project root and per-plugin)
|
||||
- Breaking changes MUST increment MAJOR version for plugins and be clearly documented
|
||||
- Release process MUST follow the documented procedure in `CLAUDE.md` including: updating changelogs, version files, running `make releaseInfo`, using `[release]` tag in commit message
|
||||
|
||||
**Rationale**: Predictable versioning enables users to understand compatibility and stability expectations. Calendar-based versioning for the main project makes release timing transparent, while semantic versioning for plugins enables clear communication of breaking changes.
|
||||
|
||||
### VII. Groovy Idioms and Code Standards
|
||||
|
||||
Code MUST follow Groovy best practices and Nextflow conventions:
|
||||
|
||||
- Use Groovy idioms (closures, operator overloading, DSL builders) appropriately
|
||||
- Follow existing code patterns and conventions from similar modules
|
||||
- Leverage Groovy's dynamic capabilities judiciously without sacrificing type safety where beneficial
|
||||
- Use Groovy's `@CompileStatic` where performance is critical or type safety is desired
|
||||
- AST transformations (in `modules/nextflow`) MUST be well-documented due to their compile-time magic
|
||||
- Code MUST be formatted consistently (consider CodeNarc configuration in `gradle/codenarc.groovy`)
|
||||
|
||||
**Rationale**: Groovy enables powerful DSL capabilities that make Nextflow's language expressive, but requires discipline to maintain readability and debuggability. Consistency across the large codebase improves maintainability.
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Build and Development Process
|
||||
|
||||
- **Build tool**: Gradle with wrapper (`./gradlew`) is the authoritative build system
|
||||
- **Quick commands**: Makefile provides convenience targets (`make compile`, `make test`, `make assemble`, `make check`, `make clean`)
|
||||
- **Development testing**: Use `./launch.sh run script.nf` for testing changes against real workflows without full installation
|
||||
- **Local installation**: `make install` publishes to Maven local for integration testing
|
||||
- **Dependency management**: All dependencies MUST be declared in `build.gradle` with explicit versions; use `make deps` to analyze dependency trees
|
||||
|
||||
### Git Workflow
|
||||
|
||||
- **Branch management**: Work on feature branches, never commit directly to `master`
|
||||
- **Commit sign-off**: Always use `git commit -s` to add DCO sign-off
|
||||
- **CI control tags**: Use special commit message tags to control CI behavior:
|
||||
- `[ci skip]` - Skip CI tests entirely
|
||||
- `[ci fast]` - Run only unit tests, skip integration tests
|
||||
- `[e2e stage]` - Run end-to-end tests against Seqera platform staging environment
|
||||
- `[e2e prod]` - Run end-to-end tests against production platform
|
||||
- `[release]` - Trigger release automation
|
||||
- **Pull requests**: Must pass all CI checks, require code review, and have DCO verification
|
||||
|
||||
### Architecture Decision Records (ADRs)
|
||||
|
||||
- Significant structural and technical decisions MUST be documented as ADRs in the `adr/` directory
|
||||
- ADRs MUST follow the template format: date prefix + descriptive name (e.g., `20251114-module-system.md`)
|
||||
- ADRs provide historical context for why architectural decisions were made
|
||||
- When changing fundamental architecture, review existing ADRs and create new ones documenting the rationale
|
||||
|
||||
## Quality Standards
|
||||
|
||||
### Code Review Requirements
|
||||
|
||||
- All changes MUST go through pull request review
|
||||
- Reviewers MUST verify:
|
||||
- Tests are included and passing
|
||||
- Code follows Groovy idioms and project conventions
|
||||
- License headers are present
|
||||
- DCO sign-off is present
|
||||
- Changes align with modular architecture principles
|
||||
- Breaking changes are appropriately versioned and documented
|
||||
|
||||
### Testing Gates
|
||||
|
||||
- `make test` MUST pass before committing locally
|
||||
- All CI tests MUST pass before merging
|
||||
- Integration tests MUST be run for changes affecting workflow execution
|
||||
- Cloud validation tests MUST be run before releases touching cloud provider plugins
|
||||
- Smoke tests enable rapid iteration but MUST NOT replace full test execution
|
||||
|
||||
### Performance and Compatibility
|
||||
|
||||
- Target platform: Java 17 runtime compatibility (development uses Java 21 toolchain)
|
||||
- Performance-critical paths SHOULD be profiled and optimized
|
||||
- Memory usage SHOULD be monitored for large-scale workflows
|
||||
- Backward compatibility MUST be maintained for existing DSL features unless a new DSL version is introduced
|
||||
|
||||
## Governance
|
||||
|
||||
### Amendment Process
|
||||
|
||||
This constitution supersedes all other development practices. Amendments require:
|
||||
|
||||
1. **Proposal**: Submit amendment proposal via GitHub issue or pull request
|
||||
2. **Discussion**: Community discussion period (minimum 1 week for major changes)
|
||||
3. **Approval**: Approval from core maintainers
|
||||
4. **Documentation**: Update this constitution with version bump following semantic versioning:
|
||||
- **MAJOR**: Backward incompatible governance changes, principle removal/redefinition
|
||||
- **MINOR**: New principle added or materially expanded guidance
|
||||
- **PATCH**: Clarifications, wording improvements, typo fixes
|
||||
|
||||
### Compliance and Review
|
||||
|
||||
- All pull requests and code reviews MUST verify compliance with these principles
|
||||
- Deviations from principles MUST be explicitly justified in PR description
|
||||
- Complexity additions MUST be justified against the "simplicity first" principle
|
||||
- Constitution compliance is enforced through code review and CI automation where possible
|
||||
|
||||
### Ratification and Version History
|
||||
|
||||
**Version**: 1.0.0 | **Ratified**: 2025-11-17 | **Last Amended**: 2025-11-17
|
||||
|
||||
This constitution was derived from the Nextflow project's documented practices in `CLAUDE.md`, `CONTRIBUTING.md`, and the project's existing architectural patterns. It codifies the development principles that have made Nextflow a successful scientific workflow management system.
|
||||
166
nextflow/.specify/scripts/bash/check-prerequisites.sh
Executable file
166
nextflow/.specify/scripts/bash/check-prerequisites.sh
Executable file
@@ -0,0 +1,166 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Consolidated prerequisite checking script
|
||||
#
|
||||
# This script provides unified prerequisite checking for Spec-Driven Development workflow.
|
||||
# It replaces the functionality previously spread across multiple scripts.
|
||||
#
|
||||
# Usage: ./check-prerequisites.sh [OPTIONS]
|
||||
#
|
||||
# OPTIONS:
|
||||
# --json Output in JSON format
|
||||
# --require-tasks Require tasks.md to exist (for implementation phase)
|
||||
# --include-tasks Include tasks.md in AVAILABLE_DOCS list
|
||||
# --paths-only Only output path variables (no validation)
|
||||
# --help, -h Show help message
|
||||
#
|
||||
# OUTPUTS:
|
||||
# JSON mode: {"FEATURE_DIR":"...", "AVAILABLE_DOCS":["..."]}
|
||||
# Text mode: FEATURE_DIR:... \n AVAILABLE_DOCS: \n ✓/✗ file.md
|
||||
# Paths only: REPO_ROOT: ... \n BRANCH: ... \n FEATURE_DIR: ... etc.
|
||||
|
||||
set -e
|
||||
|
||||
# Parse command line arguments
|
||||
JSON_MODE=false
|
||||
REQUIRE_TASKS=false
|
||||
INCLUDE_TASKS=false
|
||||
PATHS_ONLY=false
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--json)
|
||||
JSON_MODE=true
|
||||
;;
|
||||
--require-tasks)
|
||||
REQUIRE_TASKS=true
|
||||
;;
|
||||
--include-tasks)
|
||||
INCLUDE_TASKS=true
|
||||
;;
|
||||
--paths-only)
|
||||
PATHS_ONLY=true
|
||||
;;
|
||||
--help|-h)
|
||||
cat << 'EOF'
|
||||
Usage: check-prerequisites.sh [OPTIONS]
|
||||
|
||||
Consolidated prerequisite checking for Spec-Driven Development workflow.
|
||||
|
||||
OPTIONS:
|
||||
--json Output in JSON format
|
||||
--require-tasks Require tasks.md to exist (for implementation phase)
|
||||
--include-tasks Include tasks.md in AVAILABLE_DOCS list
|
||||
--paths-only Only output path variables (no prerequisite validation)
|
||||
--help, -h Show this help message
|
||||
|
||||
EXAMPLES:
|
||||
# Check task prerequisites (plan.md required)
|
||||
./check-prerequisites.sh --json
|
||||
|
||||
# Check implementation prerequisites (plan.md + tasks.md required)
|
||||
./check-prerequisites.sh --json --require-tasks --include-tasks
|
||||
|
||||
# Get feature paths only (no validation)
|
||||
./check-prerequisites.sh --paths-only
|
||||
|
||||
EOF
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: Unknown option '$arg'. Use --help for usage information." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Source common functions
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
# Get feature paths and validate branch
|
||||
eval $(get_feature_paths)
|
||||
check_feature_branch "$CURRENT_BRANCH" "$HAS_GIT" || exit 1
|
||||
|
||||
# If paths-only mode, output paths and exit (support JSON + paths-only combined)
|
||||
if $PATHS_ONLY; then
|
||||
if $JSON_MODE; then
|
||||
# Minimal JSON paths payload (no validation performed)
|
||||
printf '{"REPO_ROOT":"%s","BRANCH":"%s","FEATURE_DIR":"%s","FEATURE_SPEC":"%s","IMPL_PLAN":"%s","TASKS":"%s"}\n' \
|
||||
"$REPO_ROOT" "$CURRENT_BRANCH" "$FEATURE_DIR" "$FEATURE_SPEC" "$IMPL_PLAN" "$TASKS"
|
||||
else
|
||||
echo "REPO_ROOT: $REPO_ROOT"
|
||||
echo "BRANCH: $CURRENT_BRANCH"
|
||||
echo "FEATURE_DIR: $FEATURE_DIR"
|
||||
echo "FEATURE_SPEC: $FEATURE_SPEC"
|
||||
echo "IMPL_PLAN: $IMPL_PLAN"
|
||||
echo "TASKS: $TASKS"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Validate required directories and files
|
||||
if [[ ! -d "$FEATURE_DIR" ]]; then
|
||||
echo "ERROR: Feature directory not found: $FEATURE_DIR" >&2
|
||||
echo "Run /speckit.specify first to create the feature structure." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -f "$IMPL_PLAN" ]]; then
|
||||
echo "ERROR: plan.md not found in $FEATURE_DIR" >&2
|
||||
echo "Run /speckit.plan first to create the implementation plan." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check for tasks.md if required
|
||||
if $REQUIRE_TASKS && [[ ! -f "$TASKS" ]]; then
|
||||
echo "ERROR: tasks.md not found in $FEATURE_DIR" >&2
|
||||
echo "Run /speckit.tasks first to create the task list." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Build list of available documents
|
||||
docs=()
|
||||
|
||||
# Always check these optional docs
|
||||
[[ -f "$RESEARCH" ]] && docs+=("research.md")
|
||||
[[ -f "$DATA_MODEL" ]] && docs+=("data-model.md")
|
||||
|
||||
# Check contracts directory (only if it exists and has files)
|
||||
if [[ -d "$CONTRACTS_DIR" ]] && [[ -n "$(ls -A "$CONTRACTS_DIR" 2>/dev/null)" ]]; then
|
||||
docs+=("contracts/")
|
||||
fi
|
||||
|
||||
[[ -f "$QUICKSTART" ]] && docs+=("quickstart.md")
|
||||
|
||||
# Include tasks.md if requested and it exists
|
||||
if $INCLUDE_TASKS && [[ -f "$TASKS" ]]; then
|
||||
docs+=("tasks.md")
|
||||
fi
|
||||
|
||||
# Output results
|
||||
if $JSON_MODE; then
|
||||
# Build JSON array of documents
|
||||
if [[ ${#docs[@]} -eq 0 ]]; then
|
||||
json_docs="[]"
|
||||
else
|
||||
json_docs=$(printf '"%s",' "${docs[@]}")
|
||||
json_docs="[${json_docs%,}]"
|
||||
fi
|
||||
|
||||
printf '{"FEATURE_DIR":"%s","AVAILABLE_DOCS":%s}\n' "$FEATURE_DIR" "$json_docs"
|
||||
else
|
||||
# Text output
|
||||
echo "FEATURE_DIR:$FEATURE_DIR"
|
||||
echo "AVAILABLE_DOCS:"
|
||||
|
||||
# Show status of each potential document
|
||||
check_file "$RESEARCH" "research.md"
|
||||
check_file "$DATA_MODEL" "data-model.md"
|
||||
check_dir "$CONTRACTS_DIR" "contracts/"
|
||||
check_file "$QUICKSTART" "quickstart.md"
|
||||
|
||||
if $INCLUDE_TASKS; then
|
||||
check_file "$TASKS" "tasks.md"
|
||||
fi
|
||||
fi
|
||||
157
nextflow/.specify/scripts/bash/common.sh
Executable file
157
nextflow/.specify/scripts/bash/common.sh
Executable file
@@ -0,0 +1,157 @@
|
||||
#!/usr/bin/env bash
|
||||
# Common functions and variables for all scripts
|
||||
|
||||
# Get repository root, with fallback for non-git repositories
|
||||
get_repo_root() {
|
||||
if git rev-parse --show-toplevel >/dev/null 2>&1; then
|
||||
git rev-parse --show-toplevel
|
||||
else
|
||||
# Fall back to script location for non-git repos
|
||||
local script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
(cd "$script_dir/../../.." && pwd)
|
||||
fi
|
||||
}
|
||||
|
||||
# Get current branch, with fallback for non-git repositories
|
||||
get_current_branch() {
|
||||
# First check if SPECIFY_FEATURE environment variable is set
|
||||
if [[ -n "${SPECIFY_FEATURE:-}" ]]; then
|
||||
echo "$SPECIFY_FEATURE"
|
||||
return
|
||||
fi
|
||||
|
||||
# Then check git if available
|
||||
if git rev-parse --abbrev-ref HEAD >/dev/null 2>&1; then
|
||||
git rev-parse --abbrev-ref HEAD
|
||||
return
|
||||
fi
|
||||
|
||||
# For non-git repos, try to find the latest feature directory
|
||||
local repo_root=$(get_repo_root)
|
||||
local specs_dir="$repo_root/specs"
|
||||
|
||||
if [[ -d "$specs_dir" ]]; then
|
||||
local latest_feature=""
|
||||
local highest=0
|
||||
|
||||
for dir in "$specs_dir"/*; do
|
||||
if [[ -d "$dir" ]]; then
|
||||
local dirname=$(basename "$dir")
|
||||
if [[ "$dirname" =~ ^([0-9]{6})- ]]; then
|
||||
local number=${BASH_REMATCH[1]}
|
||||
number=$((10#$number))
|
||||
if [[ "$number" -gt "$highest" ]]; then
|
||||
highest=$number
|
||||
latest_feature=$dirname
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ -n "$latest_feature" ]]; then
|
||||
echo "$latest_feature"
|
||||
return
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "main" # Final fallback
|
||||
}
|
||||
|
||||
# Check if we have git available
|
||||
has_git() {
|
||||
git rev-parse --show-toplevel >/dev/null 2>&1
|
||||
}
|
||||
|
||||
check_feature_branch() {
|
||||
local branch="$1"
|
||||
local has_git_repo="$2"
|
||||
|
||||
# For non-git repos, we can't enforce branch naming but still provide output
|
||||
if [[ "$has_git_repo" != "true" ]]; then
|
||||
echo "[specify] Warning: Git repository not detected; skipped branch validation" >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Accept date-based format (YYMMDD-feature)
|
||||
if [[ ! "$branch" =~ ^[0-9]{6}- ]]; then
|
||||
echo "ERROR: Not on a feature branch. Current branch: $branch" >&2
|
||||
echo "Feature branches should be named like: YYMMDD-feature-name (e.g., 251030-nextflow-modules)" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
get_feature_dir() { echo "$1/specs/$2"; }
|
||||
|
||||
# Find feature directory by numeric prefix instead of exact branch match
|
||||
# This allows multiple branches to work on the same spec (e.g., 251030-fix-bug, 251030-add-feature)
|
||||
find_feature_dir_by_prefix() {
|
||||
local repo_root="$1"
|
||||
local branch_name="$2"
|
||||
local specs_dir="$repo_root/specs"
|
||||
|
||||
# Extract numeric prefix from branch (e.g., "251030" from "251030-whatever")
|
||||
if [[ ! "$branch_name" =~ ^([0-9]{6})- ]]; then
|
||||
# If branch doesn't have numeric prefix, fall back to exact match
|
||||
echo "$specs_dir/$branch_name"
|
||||
return
|
||||
fi
|
||||
|
||||
local prefix="${BASH_REMATCH[1]}"
|
||||
|
||||
# Search for directories in specs/ that start with this prefix
|
||||
local matches=()
|
||||
if [[ -d "$specs_dir" ]]; then
|
||||
for dir in "$specs_dir"/"$prefix"-*; do
|
||||
if [[ -d "$dir" ]]; then
|
||||
matches+=("$(basename "$dir")")
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# Handle results
|
||||
if [[ ${#matches[@]} -eq 0 ]]; then
|
||||
# No match found - return the branch name path (will fail later with clear error)
|
||||
echo "$specs_dir/$branch_name"
|
||||
elif [[ ${#matches[@]} -eq 1 ]]; then
|
||||
# Exactly one match - perfect!
|
||||
echo "$specs_dir/${matches[0]}"
|
||||
else
|
||||
# Multiple matches - this shouldn't happen with proper naming convention
|
||||
echo "ERROR: Multiple spec directories found with prefix '$prefix': ${matches[*]}" >&2
|
||||
echo "Please ensure only one spec directory exists per numeric prefix." >&2
|
||||
echo "$specs_dir/$branch_name" # Return something to avoid breaking the script
|
||||
fi
|
||||
}
|
||||
|
||||
get_feature_paths() {
|
||||
local repo_root=$(get_repo_root)
|
||||
local current_branch=$(get_current_branch)
|
||||
local has_git_repo="false"
|
||||
|
||||
if has_git; then
|
||||
has_git_repo="true"
|
||||
fi
|
||||
|
||||
# Use prefix-based lookup to support multiple branches per spec
|
||||
local feature_dir=$(find_feature_dir_by_prefix "$repo_root" "$current_branch")
|
||||
|
||||
cat <<EOF
|
||||
REPO_ROOT='$repo_root'
|
||||
CURRENT_BRANCH='$current_branch'
|
||||
HAS_GIT='$has_git_repo'
|
||||
FEATURE_DIR='$feature_dir'
|
||||
FEATURE_SPEC='$feature_dir/spec.md'
|
||||
IMPL_PLAN='$feature_dir/plan.md'
|
||||
TASKS='$feature_dir/tasks.md'
|
||||
RESEARCH='$feature_dir/research.md'
|
||||
DATA_MODEL='$feature_dir/data-model.md'
|
||||
QUICKSTART='$feature_dir/quickstart.md'
|
||||
CONTRACTS_DIR='$feature_dir/contracts'
|
||||
EOF
|
||||
}
|
||||
|
||||
check_file() { [[ -f "$1" ]] && echo " ✓ $2" || echo " ✗ $2"; }
|
||||
check_dir() { [[ -d "$1" && -n $(ls -A "$1" 2>/dev/null) ]] && echo " ✓ $2" || echo " ✗ $2"; }
|
||||
|
||||
200
nextflow/.specify/scripts/bash/create-new-feature.sh
Executable file
200
nextflow/.specify/scripts/bash/create-new-feature.sh
Executable file
@@ -0,0 +1,200 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -e
|
||||
|
||||
JSON_MODE=false
|
||||
SHORT_NAME=""
|
||||
ARGS=()
|
||||
i=1
|
||||
while [ $i -le $# ]; do
|
||||
arg="${!i}"
|
||||
case "$arg" in
|
||||
--json)
|
||||
JSON_MODE=true
|
||||
;;
|
||||
--short-name)
|
||||
if [ $((i + 1)) -gt $# ]; then
|
||||
echo 'Error: --short-name requires a value' >&2
|
||||
exit 1
|
||||
fi
|
||||
i=$((i + 1))
|
||||
next_arg="${!i}"
|
||||
# Check if the next argument is another option (starts with --)
|
||||
if [[ "$next_arg" == --* ]]; then
|
||||
echo 'Error: --short-name requires a value' >&2
|
||||
exit 1
|
||||
fi
|
||||
SHORT_NAME="$next_arg"
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: $0 [--json] [--short-name <name>] <feature_description>"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --json Output in JSON format"
|
||||
echo " --short-name <name> Provide a custom short name (2-4 words) for the branch"
|
||||
echo " --help, -h Show this help message"
|
||||
echo ""
|
||||
echo "Branch naming: Uses YYMMDD-<feature-name> format (e.g., 251023-user-auth)"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 'Add user authentication system' --short-name 'user-auth'"
|
||||
echo " # Creates: 251023-user-auth (if run on Oct 23, 2025)"
|
||||
echo ""
|
||||
echo " $0 'Implement OAuth2 integration for API'"
|
||||
echo " # Creates: 251023-oauth2-integration-api"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$arg")
|
||||
;;
|
||||
esac
|
||||
i=$((i + 1))
|
||||
done
|
||||
|
||||
FEATURE_DESCRIPTION="${ARGS[*]}"
|
||||
if [ -z "$FEATURE_DESCRIPTION" ]; then
|
||||
echo "Usage: $0 [--json] [--short-name <name>] <feature_description>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Function to find the repository root by searching for existing project markers
|
||||
find_repo_root() {
|
||||
local dir="$1"
|
||||
while [ "$dir" != "/" ]; do
|
||||
if [ -d "$dir/.git" ] || [ -d "$dir/.specify" ]; then
|
||||
echo "$dir"
|
||||
return 0
|
||||
fi
|
||||
dir="$(dirname "$dir")"
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Resolve repository root. Prefer git information when available, but fall back
|
||||
# to searching for repository markers so the workflow still functions in repositories that
|
||||
# were initialised with --no-git.
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
if git rev-parse --show-toplevel >/dev/null 2>&1; then
|
||||
REPO_ROOT=$(git rev-parse --show-toplevel)
|
||||
HAS_GIT=true
|
||||
else
|
||||
REPO_ROOT="$(find_repo_root "$SCRIPT_DIR")"
|
||||
if [ -z "$REPO_ROOT" ]; then
|
||||
echo "Error: Could not determine repository root. Please run this script from within the repository." >&2
|
||||
exit 1
|
||||
fi
|
||||
HAS_GIT=false
|
||||
fi
|
||||
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
SPECS_DIR="$REPO_ROOT/specs"
|
||||
mkdir -p "$SPECS_DIR"
|
||||
|
||||
# Use YYMMDD format for feature numbering (date-based prefix)
|
||||
FEATURE_NUM=$(date +%y%m%d)
|
||||
|
||||
# Function to generate branch name with stop word filtering and length filtering
|
||||
generate_branch_name() {
|
||||
local description="$1"
|
||||
|
||||
# Common stop words to filter out
|
||||
local stop_words="^(i|a|an|the|to|for|of|in|on|at|by|with|from|is|are|was|were|be|been|being|have|has|had|do|does|did|will|would|should|could|can|may|might|must|shall|this|that|these|those|my|your|our|their|want|need|add|get|set)$"
|
||||
|
||||
# Convert to lowercase and split into words
|
||||
local clean_name=$(echo "$description" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/ /g')
|
||||
|
||||
# Filter words: remove stop words and words shorter than 3 chars (unless they're uppercase acronyms in original)
|
||||
local meaningful_words=()
|
||||
for word in $clean_name; do
|
||||
# Skip empty words
|
||||
[ -z "$word" ] && continue
|
||||
|
||||
# Keep words that are NOT stop words AND (length >= 3 OR are potential acronyms)
|
||||
if ! echo "$word" | grep -qiE "$stop_words"; then
|
||||
if [ ${#word} -ge 3 ]; then
|
||||
meaningful_words+=("$word")
|
||||
elif echo "$description" | grep -q "\b${word^^}\b"; then
|
||||
# Keep short words if they appear as uppercase in original (likely acronyms)
|
||||
meaningful_words+=("$word")
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
# If we have meaningful words, use first 3-4 of them
|
||||
if [ ${#meaningful_words[@]} -gt 0 ]; then
|
||||
local max_words=3
|
||||
if [ ${#meaningful_words[@]} -eq 4 ]; then max_words=4; fi
|
||||
|
||||
local result=""
|
||||
local count=0
|
||||
for word in "${meaningful_words[@]}"; do
|
||||
if [ $count -ge $max_words ]; then break; fi
|
||||
if [ -n "$result" ]; then result="$result-"; fi
|
||||
result="$result$word"
|
||||
count=$((count + 1))
|
||||
done
|
||||
echo "$result"
|
||||
else
|
||||
# Fallback to original logic if no meaningful words found
|
||||
echo "$description" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/-\+/-/g' | sed 's/^-//' | sed 's/-$//' | tr '-' '\n' | grep -v '^$' | head -3 | tr '\n' '-' | sed 's/-$//'
|
||||
fi
|
||||
}
|
||||
|
||||
# Generate branch name
|
||||
if [ -n "$SHORT_NAME" ]; then
|
||||
# Use provided short name, just clean it up
|
||||
BRANCH_SUFFIX=$(echo "$SHORT_NAME" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/-\+/-/g' | sed 's/^-//' | sed 's/-$//')
|
||||
else
|
||||
# Generate from description with smart filtering
|
||||
BRANCH_SUFFIX=$(generate_branch_name "$FEATURE_DESCRIPTION")
|
||||
fi
|
||||
|
||||
BRANCH_NAME="${FEATURE_NUM}-${BRANCH_SUFFIX}"
|
||||
|
||||
# GitHub enforces a 244-byte limit on branch names
|
||||
# Validate and truncate if necessary
|
||||
MAX_BRANCH_LENGTH=244
|
||||
if [ ${#BRANCH_NAME} -gt $MAX_BRANCH_LENGTH ]; then
|
||||
# Calculate how much we need to trim from suffix
|
||||
# Account for: feature number (3) + hyphen (1) = 4 chars
|
||||
MAX_SUFFIX_LENGTH=$((MAX_BRANCH_LENGTH - 4))
|
||||
|
||||
# Truncate suffix at word boundary if possible
|
||||
TRUNCATED_SUFFIX=$(echo "$BRANCH_SUFFIX" | cut -c1-$MAX_SUFFIX_LENGTH)
|
||||
# Remove trailing hyphen if truncation created one
|
||||
TRUNCATED_SUFFIX=$(echo "$TRUNCATED_SUFFIX" | sed 's/-$//')
|
||||
|
||||
ORIGINAL_BRANCH_NAME="$BRANCH_NAME"
|
||||
BRANCH_NAME="${FEATURE_NUM}-${TRUNCATED_SUFFIX}"
|
||||
|
||||
>&2 echo "[specify] Warning: Branch name exceeded GitHub's 244-byte limit"
|
||||
>&2 echo "[specify] Original: $ORIGINAL_BRANCH_NAME (${#ORIGINAL_BRANCH_NAME} bytes)"
|
||||
>&2 echo "[specify] Truncated to: $BRANCH_NAME (${#BRANCH_NAME} bytes)"
|
||||
fi
|
||||
|
||||
if [ "$HAS_GIT" = true ]; then
|
||||
git checkout -b "$BRANCH_NAME"
|
||||
else
|
||||
>&2 echo "[specify] Warning: Git repository not detected; skipped branch creation for $BRANCH_NAME"
|
||||
fi
|
||||
|
||||
FEATURE_DIR="$SPECS_DIR/$BRANCH_NAME"
|
||||
mkdir -p "$FEATURE_DIR"
|
||||
|
||||
TEMPLATE="$REPO_ROOT/.specify/templates/spec-template.md"
|
||||
SPEC_FILE="$FEATURE_DIR/spec.md"
|
||||
if [ -f "$TEMPLATE" ]; then cp "$TEMPLATE" "$SPEC_FILE"; else touch "$SPEC_FILE"; fi
|
||||
|
||||
# Set the SPECIFY_FEATURE environment variable for the current session
|
||||
export SPECIFY_FEATURE="$BRANCH_NAME"
|
||||
|
||||
if $JSON_MODE; then
|
||||
printf '{"BRANCH_NAME":"%s","SPEC_FILE":"%s","FEATURE_NUM":"%s"}\n' "$BRANCH_NAME" "$SPEC_FILE" "$FEATURE_NUM"
|
||||
else
|
||||
echo "BRANCH_NAME: $BRANCH_NAME"
|
||||
echo "SPEC_FILE: $SPEC_FILE"
|
||||
echo "FEATURE_NUM: $FEATURE_NUM"
|
||||
echo "SPECIFY_FEATURE environment variable set to: $BRANCH_NAME"
|
||||
fi
|
||||
61
nextflow/.specify/scripts/bash/setup-plan.sh
Executable file
61
nextflow/.specify/scripts/bash/setup-plan.sh
Executable file
@@ -0,0 +1,61 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -e
|
||||
|
||||
# Parse command line arguments
|
||||
JSON_MODE=false
|
||||
ARGS=()
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--json)
|
||||
JSON_MODE=true
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: $0 [--json]"
|
||||
echo " --json Output results in JSON format"
|
||||
echo " --help Show this help message"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$arg")
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Get script directory and load common functions
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
# Get all paths and variables from common functions
|
||||
eval $(get_feature_paths)
|
||||
|
||||
# Check if we're on a proper feature branch (only for git repos)
|
||||
check_feature_branch "$CURRENT_BRANCH" "$HAS_GIT" || exit 1
|
||||
|
||||
# Ensure the feature directory exists
|
||||
mkdir -p "$FEATURE_DIR"
|
||||
|
||||
# Copy plan template if it exists
|
||||
TEMPLATE="$REPO_ROOT/.specify/templates/plan-template.md"
|
||||
if [[ -f "$TEMPLATE" ]]; then
|
||||
cp "$TEMPLATE" "$IMPL_PLAN"
|
||||
echo "Copied plan template to $IMPL_PLAN"
|
||||
else
|
||||
echo "Warning: Plan template not found at $TEMPLATE"
|
||||
# Create a basic plan file if template doesn't exist
|
||||
touch "$IMPL_PLAN"
|
||||
fi
|
||||
|
||||
# Output results
|
||||
if $JSON_MODE; then
|
||||
printf '{"FEATURE_SPEC":"%s","IMPL_PLAN":"%s","SPECS_DIR":"%s","BRANCH":"%s","HAS_GIT":"%s"}\n' \
|
||||
"$FEATURE_SPEC" "$IMPL_PLAN" "$FEATURE_DIR" "$CURRENT_BRANCH" "$HAS_GIT"
|
||||
else
|
||||
echo "FEATURE_SPEC: $FEATURE_SPEC"
|
||||
echo "IMPL_PLAN: $IMPL_PLAN"
|
||||
echo "SPECS_DIR: $FEATURE_DIR"
|
||||
echo "BRANCH: $CURRENT_BRANCH"
|
||||
echo "HAS_GIT: $HAS_GIT"
|
||||
fi
|
||||
|
||||
772
nextflow/.specify/scripts/bash/update-agent-context.sh
Executable file
772
nextflow/.specify/scripts/bash/update-agent-context.sh
Executable file
@@ -0,0 +1,772 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Update agent context files with information from plan.md
|
||||
#
|
||||
# This script maintains AI agent context files by parsing feature specifications
|
||||
# and updating agent-specific configuration files with project information.
|
||||
#
|
||||
# MAIN FUNCTIONS:
|
||||
# 1. Environment Validation
|
||||
# - Verifies git repository structure and branch information
|
||||
# - Checks for required plan.md files and templates
|
||||
# - Validates file permissions and accessibility
|
||||
#
|
||||
# 2. Plan Data Extraction
|
||||
# - Parses plan.md files to extract project metadata
|
||||
# - Identifies language/version, frameworks, databases, and project types
|
||||
# - Handles missing or incomplete specification data gracefully
|
||||
#
|
||||
# 3. Agent File Management
|
||||
# - Creates new agent context files from templates when needed
|
||||
# - Updates existing agent files with new project information
|
||||
# - Preserves manual additions and custom configurations
|
||||
# - Supports multiple AI agent formats and directory structures
|
||||
#
|
||||
# 4. Content Generation
|
||||
# - Generates language-specific build/test commands
|
||||
# - Creates appropriate project directory structures
|
||||
# - Updates technology stacks and recent changes sections
|
||||
# - Maintains consistent formatting and timestamps
|
||||
#
|
||||
# 5. Multi-Agent Support
|
||||
# - Handles agent-specific file paths and naming conventions
|
||||
# - Supports: Claude, Gemini, Copilot, Cursor, Qwen, opencode, Codex, Windsurf, Kilo Code, Auggie CLI, Roo Code, CodeBuddy CLI, Amp, or Amazon Q Developer CLI
|
||||
# - Can update single agents or all existing agent files
|
||||
# - Creates default Claude file if no agent files exist
|
||||
#
|
||||
# Usage: ./update-agent-context.sh [agent_type]
|
||||
# Agent types: claude|gemini|copilot|cursor-agent|qwen|opencode|codex|windsurf|kilocode|auggie|q
|
||||
# Leave empty to update all existing agent files
|
||||
|
||||
set -e
|
||||
|
||||
# Enable strict error handling
|
||||
set -u
|
||||
set -o pipefail
|
||||
|
||||
#==============================================================================
|
||||
# Configuration and Global Variables
|
||||
#==============================================================================
|
||||
|
||||
# Get script directory and load common functions
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
# Get all paths and variables from common functions
|
||||
eval $(get_feature_paths)
|
||||
|
||||
NEW_PLAN="$IMPL_PLAN" # Alias for compatibility with existing code
|
||||
AGENT_TYPE="${1:-}"
|
||||
|
||||
# Agent-specific file paths
|
||||
CLAUDE_FILE="$REPO_ROOT/CLAUDE.md"
|
||||
GEMINI_FILE="$REPO_ROOT/GEMINI.md"
|
||||
COPILOT_FILE="$REPO_ROOT/.github/copilot-instructions.md"
|
||||
CURSOR_FILE="$REPO_ROOT/.cursor/rules/specify-rules.mdc"
|
||||
QWEN_FILE="$REPO_ROOT/QWEN.md"
|
||||
AGENTS_FILE="$REPO_ROOT/AGENTS.md"
|
||||
WINDSURF_FILE="$REPO_ROOT/.windsurf/rules/specify-rules.md"
|
||||
KILOCODE_FILE="$REPO_ROOT/.kilocode/rules/specify-rules.md"
|
||||
AUGGIE_FILE="$REPO_ROOT/.augment/rules/specify-rules.md"
|
||||
ROO_FILE="$REPO_ROOT/.roo/rules/specify-rules.md"
|
||||
CODEBUDDY_FILE="$REPO_ROOT/CODEBUDDY.md"
|
||||
AMP_FILE="$REPO_ROOT/AGENTS.md"
|
||||
Q_FILE="$REPO_ROOT/AGENTS.md"
|
||||
|
||||
# Template file
|
||||
TEMPLATE_FILE="$REPO_ROOT/.specify/templates/agent-file-template.md"
|
||||
|
||||
# Global variables for parsed plan data
|
||||
NEW_LANG=""
|
||||
NEW_FRAMEWORK=""
|
||||
NEW_DB=""
|
||||
NEW_PROJECT_TYPE=""
|
||||
|
||||
#==============================================================================
|
||||
# Utility Functions
|
||||
#==============================================================================
|
||||
|
||||
log_info() {
|
||||
echo "INFO: $1"
|
||||
}
|
||||
|
||||
log_success() {
|
||||
echo "✓ $1"
|
||||
}
|
||||
|
||||
log_error() {
|
||||
echo "ERROR: $1" >&2
|
||||
}
|
||||
|
||||
log_warning() {
|
||||
echo "WARNING: $1" >&2
|
||||
}
|
||||
|
||||
# Cleanup function for temporary files
|
||||
cleanup() {
|
||||
local exit_code=$?
|
||||
rm -f /tmp/agent_update_*_$$
|
||||
rm -f /tmp/manual_additions_$$
|
||||
exit $exit_code
|
||||
}
|
||||
|
||||
# Set up cleanup trap
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
#==============================================================================
|
||||
# Validation Functions
|
||||
#==============================================================================
|
||||
|
||||
validate_environment() {
|
||||
# Check if we have a current branch/feature (git or non-git)
|
||||
if [[ -z "$CURRENT_BRANCH" ]]; then
|
||||
log_error "Unable to determine current feature"
|
||||
if [[ "$HAS_GIT" == "true" ]]; then
|
||||
log_info "Make sure you're on a feature branch"
|
||||
else
|
||||
log_info "Set SPECIFY_FEATURE environment variable or create a feature first"
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if plan.md exists
|
||||
if [[ ! -f "$NEW_PLAN" ]]; then
|
||||
log_error "No plan.md found at $NEW_PLAN"
|
||||
log_info "Make sure you're working on a feature with a corresponding spec directory"
|
||||
if [[ "$HAS_GIT" != "true" ]]; then
|
||||
log_info "Use: export SPECIFY_FEATURE=your-feature-name or create a new feature first"
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if template exists (needed for new files)
|
||||
if [[ ! -f "$TEMPLATE_FILE" ]]; then
|
||||
log_warning "Template file not found at $TEMPLATE_FILE"
|
||||
log_warning "Creating new agent files will fail"
|
||||
fi
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Plan Parsing Functions
|
||||
#==============================================================================
|
||||
|
||||
extract_plan_field() {
|
||||
local field_pattern="$1"
|
||||
local plan_file="$2"
|
||||
|
||||
grep "^\*\*${field_pattern}\*\*: " "$plan_file" 2>/dev/null | \
|
||||
head -1 | \
|
||||
sed "s|^\*\*${field_pattern}\*\*: ||" | \
|
||||
sed 's/^[ \t]*//;s/[ \t]*$//' | \
|
||||
grep -v "NEEDS CLARIFICATION" | \
|
||||
grep -v "^N/A$" || echo ""
|
||||
}
|
||||
|
||||
parse_plan_data() {
|
||||
local plan_file="$1"
|
||||
|
||||
if [[ ! -f "$plan_file" ]]; then
|
||||
log_error "Plan file not found: $plan_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ ! -r "$plan_file" ]]; then
|
||||
log_error "Plan file is not readable: $plan_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Parsing plan data from $plan_file"
|
||||
|
||||
NEW_LANG=$(extract_plan_field "Language/Version" "$plan_file")
|
||||
NEW_FRAMEWORK=$(extract_plan_field "Primary Dependencies" "$plan_file")
|
||||
NEW_DB=$(extract_plan_field "Storage" "$plan_file")
|
||||
NEW_PROJECT_TYPE=$(extract_plan_field "Project Type" "$plan_file")
|
||||
|
||||
# Log what we found
|
||||
if [[ -n "$NEW_LANG" ]]; then
|
||||
log_info "Found language: $NEW_LANG"
|
||||
else
|
||||
log_warning "No language information found in plan"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_FRAMEWORK" ]]; then
|
||||
log_info "Found framework: $NEW_FRAMEWORK"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]]; then
|
||||
log_info "Found database: $NEW_DB"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_PROJECT_TYPE" ]]; then
|
||||
log_info "Found project type: $NEW_PROJECT_TYPE"
|
||||
fi
|
||||
}
|
||||
|
||||
format_technology_stack() {
|
||||
local lang="$1"
|
||||
local framework="$2"
|
||||
local parts=()
|
||||
|
||||
# Add non-empty parts
|
||||
[[ -n "$lang" && "$lang" != "NEEDS CLARIFICATION" ]] && parts+=("$lang")
|
||||
[[ -n "$framework" && "$framework" != "NEEDS CLARIFICATION" && "$framework" != "N/A" ]] && parts+=("$framework")
|
||||
|
||||
# Join with proper formatting
|
||||
if [[ ${#parts[@]} -eq 0 ]]; then
|
||||
echo ""
|
||||
elif [[ ${#parts[@]} -eq 1 ]]; then
|
||||
echo "${parts[0]}"
|
||||
else
|
||||
# Join multiple parts with " + "
|
||||
local result="${parts[0]}"
|
||||
for ((i=1; i<${#parts[@]}; i++)); do
|
||||
result="$result + ${parts[i]}"
|
||||
done
|
||||
echo "$result"
|
||||
fi
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Template and Content Generation Functions
|
||||
#==============================================================================
|
||||
|
||||
get_project_structure() {
|
||||
local project_type="$1"
|
||||
|
||||
if [[ "$project_type" == *"web"* ]]; then
|
||||
echo "backend/\\nfrontend/\\ntests/"
|
||||
else
|
||||
echo "src/\\ntests/"
|
||||
fi
|
||||
}
|
||||
|
||||
get_commands_for_language() {
|
||||
local lang="$1"
|
||||
|
||||
case "$lang" in
|
||||
*"Python"*)
|
||||
echo "cd src && pytest && ruff check ."
|
||||
;;
|
||||
*"Rust"*)
|
||||
echo "cargo test && cargo clippy"
|
||||
;;
|
||||
*"JavaScript"*|*"TypeScript"*)
|
||||
echo "npm test \\&\\& npm run lint"
|
||||
;;
|
||||
*)
|
||||
echo "# Add commands for $lang"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
get_language_conventions() {
|
||||
local lang="$1"
|
||||
echo "$lang: Follow standard conventions"
|
||||
}
|
||||
|
||||
create_new_agent_file() {
|
||||
local target_file="$1"
|
||||
local temp_file="$2"
|
||||
local project_name="$3"
|
||||
local current_date="$4"
|
||||
|
||||
if [[ ! -f "$TEMPLATE_FILE" ]]; then
|
||||
log_error "Template not found at $TEMPLATE_FILE"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ ! -r "$TEMPLATE_FILE" ]]; then
|
||||
log_error "Template file is not readable: $TEMPLATE_FILE"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Creating new agent context file from template..."
|
||||
|
||||
if ! cp "$TEMPLATE_FILE" "$temp_file"; then
|
||||
log_error "Failed to copy template file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Replace template placeholders
|
||||
local project_structure
|
||||
project_structure=$(get_project_structure "$NEW_PROJECT_TYPE")
|
||||
|
||||
local commands
|
||||
commands=$(get_commands_for_language "$NEW_LANG")
|
||||
|
||||
local language_conventions
|
||||
language_conventions=$(get_language_conventions "$NEW_LANG")
|
||||
|
||||
# Perform substitutions with error checking using safer approach
|
||||
# Escape special characters for sed by using a different delimiter or escaping
|
||||
local escaped_lang=$(printf '%s\n' "$NEW_LANG" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
||||
local escaped_framework=$(printf '%s\n' "$NEW_FRAMEWORK" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
||||
local escaped_branch=$(printf '%s\n' "$CURRENT_BRANCH" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
||||
|
||||
# Build technology stack and recent change strings conditionally
|
||||
local tech_stack
|
||||
if [[ -n "$escaped_lang" && -n "$escaped_framework" ]]; then
|
||||
tech_stack="- $escaped_lang + $escaped_framework ($escaped_branch)"
|
||||
elif [[ -n "$escaped_lang" ]]; then
|
||||
tech_stack="- $escaped_lang ($escaped_branch)"
|
||||
elif [[ -n "$escaped_framework" ]]; then
|
||||
tech_stack="- $escaped_framework ($escaped_branch)"
|
||||
else
|
||||
tech_stack="- ($escaped_branch)"
|
||||
fi
|
||||
|
||||
local recent_change
|
||||
if [[ -n "$escaped_lang" && -n "$escaped_framework" ]]; then
|
||||
recent_change="- $escaped_branch: Added $escaped_lang + $escaped_framework"
|
||||
elif [[ -n "$escaped_lang" ]]; then
|
||||
recent_change="- $escaped_branch: Added $escaped_lang"
|
||||
elif [[ -n "$escaped_framework" ]]; then
|
||||
recent_change="- $escaped_branch: Added $escaped_framework"
|
||||
else
|
||||
recent_change="- $escaped_branch: Added"
|
||||
fi
|
||||
|
||||
local substitutions=(
|
||||
"s|\[PROJECT NAME\]|$project_name|"
|
||||
"s|\[DATE\]|$current_date|"
|
||||
"s|\[EXTRACTED FROM ALL PLAN.MD FILES\]|$tech_stack|"
|
||||
"s|\[ACTUAL STRUCTURE FROM PLANS\]|$project_structure|g"
|
||||
"s|\[ONLY COMMANDS FOR ACTIVE TECHNOLOGIES\]|$commands|"
|
||||
"s|\[LANGUAGE-SPECIFIC, ONLY FOR LANGUAGES IN USE\]|$language_conventions|"
|
||||
"s|\[LAST 3 FEATURES AND WHAT THEY ADDED\]|$recent_change|"
|
||||
)
|
||||
|
||||
for substitution in "${substitutions[@]}"; do
|
||||
if ! sed -i.bak -e "$substitution" "$temp_file"; then
|
||||
log_error "Failed to perform substitution: $substitution"
|
||||
rm -f "$temp_file" "$temp_file.bak"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Convert \n sequences to actual newlines
|
||||
newline=$(printf '\n')
|
||||
sed -i.bak2 "s/\\\\n/${newline}/g" "$temp_file"
|
||||
|
||||
# Clean up backup files
|
||||
rm -f "$temp_file.bak" "$temp_file.bak2"
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
update_existing_agent_file() {
|
||||
local target_file="$1"
|
||||
local current_date="$2"
|
||||
|
||||
log_info "Updating existing agent context file..."
|
||||
|
||||
# Use a single temporary file for atomic update
|
||||
local temp_file
|
||||
temp_file=$(mktemp) || {
|
||||
log_error "Failed to create temporary file"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Process the file in one pass
|
||||
local tech_stack=$(format_technology_stack "$NEW_LANG" "$NEW_FRAMEWORK")
|
||||
local new_tech_entries=()
|
||||
local new_change_entry=""
|
||||
|
||||
# Prepare new technology entries
|
||||
if [[ -n "$tech_stack" ]] && ! grep -q "$tech_stack" "$target_file"; then
|
||||
new_tech_entries+=("- $tech_stack ($CURRENT_BRANCH)")
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]] && [[ "$NEW_DB" != "NEEDS CLARIFICATION" ]] && ! grep -q "$NEW_DB" "$target_file"; then
|
||||
new_tech_entries+=("- $NEW_DB ($CURRENT_BRANCH)")
|
||||
fi
|
||||
|
||||
# Prepare new change entry
|
||||
if [[ -n "$tech_stack" ]]; then
|
||||
new_change_entry="- $CURRENT_BRANCH: Added $tech_stack"
|
||||
elif [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]] && [[ "$NEW_DB" != "NEEDS CLARIFICATION" ]]; then
|
||||
new_change_entry="- $CURRENT_BRANCH: Added $NEW_DB"
|
||||
fi
|
||||
|
||||
# Check if sections exist in the file
|
||||
local has_active_technologies=0
|
||||
local has_recent_changes=0
|
||||
|
||||
if grep -q "^## Active Technologies" "$target_file" 2>/dev/null; then
|
||||
has_active_technologies=1
|
||||
fi
|
||||
|
||||
if grep -q "^## Recent Changes" "$target_file" 2>/dev/null; then
|
||||
has_recent_changes=1
|
||||
fi
|
||||
|
||||
# Process file line by line
|
||||
local in_tech_section=false
|
||||
local in_changes_section=false
|
||||
local tech_entries_added=false
|
||||
local changes_entries_added=false
|
||||
local existing_changes_count=0
|
||||
local file_ended=false
|
||||
|
||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
||||
# Handle Active Technologies section
|
||||
if [[ "$line" == "## Active Technologies" ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
in_tech_section=true
|
||||
continue
|
||||
elif [[ $in_tech_section == true ]] && [[ "$line" =~ ^##[[:space:]] ]]; then
|
||||
# Add new tech entries before closing the section
|
||||
if [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
||||
tech_entries_added=true
|
||||
fi
|
||||
echo "$line" >> "$temp_file"
|
||||
in_tech_section=false
|
||||
continue
|
||||
elif [[ $in_tech_section == true ]] && [[ -z "$line" ]]; then
|
||||
# Add new tech entries before empty line in tech section
|
||||
if [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
||||
tech_entries_added=true
|
||||
fi
|
||||
echo "$line" >> "$temp_file"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Handle Recent Changes section
|
||||
if [[ "$line" == "## Recent Changes" ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
# Add new change entry right after the heading
|
||||
if [[ -n "$new_change_entry" ]]; then
|
||||
echo "$new_change_entry" >> "$temp_file"
|
||||
fi
|
||||
in_changes_section=true
|
||||
changes_entries_added=true
|
||||
continue
|
||||
elif [[ $in_changes_section == true ]] && [[ "$line" =~ ^##[[:space:]] ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
in_changes_section=false
|
||||
continue
|
||||
elif [[ $in_changes_section == true ]] && [[ "$line" == "- "* ]]; then
|
||||
# Keep only first 2 existing changes
|
||||
if [[ $existing_changes_count -lt 2 ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
((existing_changes_count++))
|
||||
fi
|
||||
continue
|
||||
fi
|
||||
|
||||
# Update timestamp
|
||||
if [[ "$line" =~ \*\*Last\ updated\*\*:.*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] ]]; then
|
||||
echo "$line" | sed "s/[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]/$current_date/" >> "$temp_file"
|
||||
else
|
||||
echo "$line" >> "$temp_file"
|
||||
fi
|
||||
done < "$target_file"
|
||||
|
||||
# Post-loop check: if we're still in the Active Technologies section and haven't added new entries
|
||||
if [[ $in_tech_section == true ]] && [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
||||
tech_entries_added=true
|
||||
fi
|
||||
|
||||
# If sections don't exist, add them at the end of the file
|
||||
if [[ $has_active_technologies -eq 0 ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
||||
echo "" >> "$temp_file"
|
||||
echo "## Active Technologies" >> "$temp_file"
|
||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
||||
tech_entries_added=true
|
||||
fi
|
||||
|
||||
if [[ $has_recent_changes -eq 0 ]] && [[ -n "$new_change_entry" ]]; then
|
||||
echo "" >> "$temp_file"
|
||||
echo "## Recent Changes" >> "$temp_file"
|
||||
echo "$new_change_entry" >> "$temp_file"
|
||||
changes_entries_added=true
|
||||
fi
|
||||
|
||||
# Move temp file to target atomically
|
||||
if ! mv "$temp_file" "$target_file"; then
|
||||
log_error "Failed to update target file"
|
||||
rm -f "$temp_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
#==============================================================================
|
||||
# Main Agent File Update Function
|
||||
#==============================================================================
|
||||
|
||||
update_agent_file() {
|
||||
local target_file="$1"
|
||||
local agent_name="$2"
|
||||
|
||||
if [[ -z "$target_file" ]] || [[ -z "$agent_name" ]]; then
|
||||
log_error "update_agent_file requires target_file and agent_name parameters"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Updating $agent_name context file: $target_file"
|
||||
|
||||
local project_name
|
||||
project_name=$(basename "$REPO_ROOT")
|
||||
local current_date
|
||||
current_date=$(date +%Y-%m-%d)
|
||||
|
||||
# Create directory if it doesn't exist
|
||||
local target_dir
|
||||
target_dir=$(dirname "$target_file")
|
||||
if [[ ! -d "$target_dir" ]]; then
|
||||
if ! mkdir -p "$target_dir"; then
|
||||
log_error "Failed to create directory: $target_dir"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ ! -f "$target_file" ]]; then
|
||||
# Create new file from template
|
||||
local temp_file
|
||||
temp_file=$(mktemp) || {
|
||||
log_error "Failed to create temporary file"
|
||||
return 1
|
||||
}
|
||||
|
||||
if create_new_agent_file "$target_file" "$temp_file" "$project_name" "$current_date"; then
|
||||
if mv "$temp_file" "$target_file"; then
|
||||
log_success "Created new $agent_name context file"
|
||||
else
|
||||
log_error "Failed to move temporary file to $target_file"
|
||||
rm -f "$temp_file"
|
||||
return 1
|
||||
fi
|
||||
else
|
||||
log_error "Failed to create new agent file"
|
||||
rm -f "$temp_file"
|
||||
return 1
|
||||
fi
|
||||
else
|
||||
# Update existing file
|
||||
if [[ ! -r "$target_file" ]]; then
|
||||
log_error "Cannot read existing file: $target_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ ! -w "$target_file" ]]; then
|
||||
log_error "Cannot write to existing file: $target_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if update_existing_agent_file "$target_file" "$current_date"; then
|
||||
log_success "Updated existing $agent_name context file"
|
||||
else
|
||||
log_error "Failed to update existing agent file"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Agent Selection and Processing
|
||||
#==============================================================================
|
||||
|
||||
update_specific_agent() {
|
||||
local agent_type="$1"
|
||||
|
||||
case "$agent_type" in
|
||||
claude)
|
||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
||||
;;
|
||||
gemini)
|
||||
update_agent_file "$GEMINI_FILE" "Gemini CLI"
|
||||
;;
|
||||
copilot)
|
||||
update_agent_file "$COPILOT_FILE" "GitHub Copilot"
|
||||
;;
|
||||
cursor-agent)
|
||||
update_agent_file "$CURSOR_FILE" "Cursor IDE"
|
||||
;;
|
||||
qwen)
|
||||
update_agent_file "$QWEN_FILE" "Qwen Code"
|
||||
;;
|
||||
opencode)
|
||||
update_agent_file "$AGENTS_FILE" "opencode"
|
||||
;;
|
||||
codex)
|
||||
update_agent_file "$AGENTS_FILE" "Codex CLI"
|
||||
;;
|
||||
windsurf)
|
||||
update_agent_file "$WINDSURF_FILE" "Windsurf"
|
||||
;;
|
||||
kilocode)
|
||||
update_agent_file "$KILOCODE_FILE" "Kilo Code"
|
||||
;;
|
||||
auggie)
|
||||
update_agent_file "$AUGGIE_FILE" "Auggie CLI"
|
||||
;;
|
||||
roo)
|
||||
update_agent_file "$ROO_FILE" "Roo Code"
|
||||
;;
|
||||
codebuddy)
|
||||
update_agent_file "$CODEBUDDY_FILE" "CodeBuddy CLI"
|
||||
;;
|
||||
amp)
|
||||
update_agent_file "$AMP_FILE" "Amp"
|
||||
;;
|
||||
q)
|
||||
update_agent_file "$Q_FILE" "Amazon Q Developer CLI"
|
||||
;;
|
||||
*)
|
||||
log_error "Unknown agent type '$agent_type'"
|
||||
log_error "Expected: claude|gemini|copilot|cursor-agent|qwen|opencode|codex|windsurf|kilocode|auggie|roo|amp|q"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
update_all_existing_agents() {
|
||||
local found_agent=false
|
||||
|
||||
# Check each possible agent file and update if it exists
|
||||
if [[ -f "$CLAUDE_FILE" ]]; then
|
||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$GEMINI_FILE" ]]; then
|
||||
update_agent_file "$GEMINI_FILE" "Gemini CLI"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$COPILOT_FILE" ]]; then
|
||||
update_agent_file "$COPILOT_FILE" "GitHub Copilot"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$CURSOR_FILE" ]]; then
|
||||
update_agent_file "$CURSOR_FILE" "Cursor IDE"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$QWEN_FILE" ]]; then
|
||||
update_agent_file "$QWEN_FILE" "Qwen Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$AGENTS_FILE" ]]; then
|
||||
update_agent_file "$AGENTS_FILE" "Codex/opencode"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$WINDSURF_FILE" ]]; then
|
||||
update_agent_file "$WINDSURF_FILE" "Windsurf"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$KILOCODE_FILE" ]]; then
|
||||
update_agent_file "$KILOCODE_FILE" "Kilo Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$AUGGIE_FILE" ]]; then
|
||||
update_agent_file "$AUGGIE_FILE" "Auggie CLI"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$ROO_FILE" ]]; then
|
||||
update_agent_file "$ROO_FILE" "Roo Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$CODEBUDDY_FILE" ]]; then
|
||||
update_agent_file "$CODEBUDDY_FILE" "CodeBuddy CLI"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$Q_FILE" ]]; then
|
||||
update_agent_file "$Q_FILE" "Amazon Q Developer CLI"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
# If no agent files exist, create a default Claude file
|
||||
if [[ "$found_agent" == false ]]; then
|
||||
log_info "No existing agent files found, creating default Claude file..."
|
||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
||||
fi
|
||||
}
|
||||
print_summary() {
|
||||
echo
|
||||
log_info "Summary of changes:"
|
||||
|
||||
if [[ -n "$NEW_LANG" ]]; then
|
||||
echo " - Added language: $NEW_LANG"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_FRAMEWORK" ]]; then
|
||||
echo " - Added framework: $NEW_FRAMEWORK"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]]; then
|
||||
echo " - Added database: $NEW_DB"
|
||||
fi
|
||||
|
||||
echo
|
||||
|
||||
log_info "Usage: $0 [claude|gemini|copilot|cursor-agent|qwen|opencode|codex|windsurf|kilocode|auggie|codebuddy|q]"
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Main Execution
|
||||
#==============================================================================
|
||||
|
||||
main() {
|
||||
# Validate environment before proceeding
|
||||
validate_environment
|
||||
|
||||
log_info "=== Updating agent context files for feature $CURRENT_BRANCH ==="
|
||||
|
||||
# Parse the plan file to extract project information
|
||||
if ! parse_plan_data "$NEW_PLAN"; then
|
||||
log_error "Failed to parse plan data"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Process based on agent type argument
|
||||
local success=true
|
||||
|
||||
if [[ -z "$AGENT_TYPE" ]]; then
|
||||
# No specific agent provided - update all existing agent files
|
||||
log_info "No agent specified, updating all existing agent files..."
|
||||
if ! update_all_existing_agents; then
|
||||
success=false
|
||||
fi
|
||||
else
|
||||
# Specific agent provided - update only that agent
|
||||
log_info "Updating specific agent: $AGENT_TYPE"
|
||||
if ! update_specific_agent "$AGENT_TYPE"; then
|
||||
success=false
|
||||
fi
|
||||
fi
|
||||
|
||||
# Print summary
|
||||
print_summary
|
||||
|
||||
if [[ "$success" == true ]]; then
|
||||
log_success "Agent context update completed successfully"
|
||||
exit 0
|
||||
else
|
||||
log_error "Agent context update completed with errors"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Execute main function if script is run directly
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
|
||||
28
nextflow/.specify/templates/agent-file-template.md
Normal file
28
nextflow/.specify/templates/agent-file-template.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# [PROJECT NAME] Development Guidelines
|
||||
|
||||
Auto-generated from all feature plans. Last updated: [DATE]
|
||||
|
||||
## Active Technologies
|
||||
|
||||
[EXTRACTED FROM ALL PLAN.MD FILES]
|
||||
|
||||
## Project Structure
|
||||
|
||||
```text
|
||||
[ACTUAL STRUCTURE FROM PLANS]
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
[ONLY COMMANDS FOR ACTIVE TECHNOLOGIES]
|
||||
|
||||
## Code Style
|
||||
|
||||
[LANGUAGE-SPECIFIC, ONLY FOR LANGUAGES IN USE]
|
||||
|
||||
## Recent Changes
|
||||
|
||||
[LAST 3 FEATURES AND WHAT THEY ADDED]
|
||||
|
||||
<!-- MANUAL ADDITIONS START -->
|
||||
<!-- MANUAL ADDITIONS END -->
|
||||
40
nextflow/.specify/templates/checklist-template.md
Normal file
40
nextflow/.specify/templates/checklist-template.md
Normal file
@@ -0,0 +1,40 @@
|
||||
# [CHECKLIST TYPE] Checklist: [FEATURE NAME]
|
||||
|
||||
**Purpose**: [Brief description of what this checklist covers]
|
||||
**Created**: [DATE]
|
||||
**Feature**: [Link to spec.md or relevant documentation]
|
||||
|
||||
**Note**: This checklist is generated by the `/speckit.checklist` command based on feature context and requirements.
|
||||
|
||||
<!--
|
||||
============================================================================
|
||||
IMPORTANT: The checklist items below are SAMPLE ITEMS for illustration only.
|
||||
|
||||
The /speckit.checklist command MUST replace these with actual items based on:
|
||||
- User's specific checklist request
|
||||
- Feature requirements from spec.md
|
||||
- Technical context from plan.md
|
||||
- Implementation details from tasks.md
|
||||
|
||||
DO NOT keep these sample items in the generated checklist file.
|
||||
============================================================================
|
||||
-->
|
||||
|
||||
## [Category 1]
|
||||
|
||||
- [ ] CHK001 First checklist item with clear action
|
||||
- [ ] CHK002 Second checklist item
|
||||
- [ ] CHK003 Third checklist item
|
||||
|
||||
## [Category 2]
|
||||
|
||||
- [ ] CHK004 Another category item
|
||||
- [ ] CHK005 Item with specific criteria
|
||||
- [ ] CHK006 Final item in this category
|
||||
|
||||
## Notes
|
||||
|
||||
- Check items off as completed: `[x]`
|
||||
- Add comments or findings inline
|
||||
- Link to relevant resources or documentation
|
||||
- Items are numbered sequentially for easy reference
|
||||
104
nextflow/.specify/templates/plan-template.md
Normal file
104
nextflow/.specify/templates/plan-template.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Implementation Plan: [FEATURE]
|
||||
|
||||
**Branch**: `[###-feature-name]` | **Date**: [DATE] | **Spec**: [link]
|
||||
**Input**: Feature specification from `/specs/[###-feature-name]/spec.md`
|
||||
|
||||
**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/commands/plan.md` for the execution workflow.
|
||||
|
||||
## Summary
|
||||
|
||||
[Extract from feature spec: primary requirement + technical approach from research]
|
||||
|
||||
## Technical Context
|
||||
|
||||
<!--
|
||||
ACTION REQUIRED: Replace the content in this section with the technical details
|
||||
for the project. The structure here is presented in advisory capacity to guide
|
||||
the iteration process.
|
||||
-->
|
||||
|
||||
**Language/Version**: [e.g., Python 3.11, Swift 5.9, Rust 1.75 or NEEDS CLARIFICATION]
|
||||
**Primary Dependencies**: [e.g., FastAPI, UIKit, LLVM or NEEDS CLARIFICATION]
|
||||
**Storage**: [if applicable, e.g., PostgreSQL, CoreData, files or N/A]
|
||||
**Testing**: [e.g., pytest, XCTest, cargo test or NEEDS CLARIFICATION]
|
||||
**Target Platform**: [e.g., Linux server, iOS 15+, WASM or NEEDS CLARIFICATION]
|
||||
**Project Type**: [single/web/mobile - determines source structure]
|
||||
**Performance Goals**: [domain-specific, e.g., 1000 req/s, 10k lines/sec, 60 fps or NEEDS CLARIFICATION]
|
||||
**Constraints**: [domain-specific, e.g., <200ms p95, <100MB memory, offline-capable or NEEDS CLARIFICATION]
|
||||
**Scale/Scope**: [domain-specific, e.g., 10k users, 1M LOC, 50 screens or NEEDS CLARIFICATION]
|
||||
|
||||
## Constitution Check
|
||||
|
||||
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
|
||||
|
||||
[Gates determined based on constitution file]
|
||||
|
||||
## Project Structure
|
||||
|
||||
### Documentation (this feature)
|
||||
|
||||
```text
|
||||
specs/[###-feature]/
|
||||
├── plan.md # This file (/speckit.plan command output)
|
||||
├── research.md # Phase 0 output (/speckit.plan command)
|
||||
├── data-model.md # Phase 1 output (/speckit.plan command)
|
||||
├── quickstart.md # Phase 1 output (/speckit.plan command)
|
||||
├── contracts/ # Phase 1 output (/speckit.plan command)
|
||||
└── tasks.md # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan)
|
||||
```
|
||||
|
||||
### Source Code (repository root)
|
||||
<!--
|
||||
ACTION REQUIRED: Replace the placeholder tree below with the concrete layout
|
||||
for this feature. Delete unused options and expand the chosen structure with
|
||||
real paths (e.g., apps/admin, packages/something). The delivered plan must
|
||||
not include Option labels.
|
||||
-->
|
||||
|
||||
```text
|
||||
# [REMOVE IF UNUSED] Option 1: Single project (DEFAULT)
|
||||
src/
|
||||
├── models/
|
||||
├── services/
|
||||
├── cli/
|
||||
└── lib/
|
||||
|
||||
tests/
|
||||
├── contract/
|
||||
├── integration/
|
||||
└── unit/
|
||||
|
||||
# [REMOVE IF UNUSED] Option 2: Web application (when "frontend" + "backend" detected)
|
||||
backend/
|
||||
├── src/
|
||||
│ ├── models/
|
||||
│ ├── services/
|
||||
│ └── api/
|
||||
└── tests/
|
||||
|
||||
frontend/
|
||||
├── src/
|
||||
│ ├── components/
|
||||
│ ├── pages/
|
||||
│ └── services/
|
||||
└── tests/
|
||||
|
||||
# [REMOVE IF UNUSED] Option 3: Mobile + API (when "iOS/Android" detected)
|
||||
api/
|
||||
└── [same as backend above]
|
||||
|
||||
ios/ or android/
|
||||
└── [platform-specific structure: feature modules, UI flows, platform tests]
|
||||
```
|
||||
|
||||
**Structure Decision**: [Document the selected structure and reference the real
|
||||
directories captured above]
|
||||
|
||||
## Complexity Tracking
|
||||
|
||||
> **Fill ONLY if Constitution Check has violations that must be justified**
|
||||
|
||||
| Violation | Why Needed | Simpler Alternative Rejected Because |
|
||||
|-----------|------------|-------------------------------------|
|
||||
| [e.g., 4th project] | [current need] | [why 3 projects insufficient] |
|
||||
| [e.g., Repository pattern] | [specific problem] | [why direct DB access insufficient] |
|
||||
115
nextflow/.specify/templates/spec-template.md
Normal file
115
nextflow/.specify/templates/spec-template.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# Feature Specification: [FEATURE NAME]
|
||||
|
||||
**Feature Branch**: `[###-feature-name]`
|
||||
**Created**: [DATE]
|
||||
**Status**: Draft
|
||||
**Input**: User description: "$ARGUMENTS"
|
||||
|
||||
## User Scenarios & Testing *(mandatory)*
|
||||
|
||||
<!--
|
||||
IMPORTANT: User stories should be PRIORITIZED as user journeys ordered by importance.
|
||||
Each user story/journey must be INDEPENDENTLY TESTABLE - meaning if you implement just ONE of them,
|
||||
you should still have a viable MVP (Minimum Viable Product) that delivers value.
|
||||
|
||||
Assign priorities (P1, P2, P3, etc.) to each story, where P1 is the most critical.
|
||||
Think of each story as a standalone slice of functionality that can be:
|
||||
- Developed independently
|
||||
- Tested independently
|
||||
- Deployed independently
|
||||
- Demonstrated to users independently
|
||||
-->
|
||||
|
||||
### User Story 1 - [Brief Title] (Priority: P1)
|
||||
|
||||
[Describe this user journey in plain language]
|
||||
|
||||
**Why this priority**: [Explain the value and why it has this priority level]
|
||||
|
||||
**Independent Test**: [Describe how this can be tested independently - e.g., "Can be fully tested by [specific action] and delivers [specific value]"]
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** [initial state], **When** [action], **Then** [expected outcome]
|
||||
2. **Given** [initial state], **When** [action], **Then** [expected outcome]
|
||||
|
||||
---
|
||||
|
||||
### User Story 2 - [Brief Title] (Priority: P2)
|
||||
|
||||
[Describe this user journey in plain language]
|
||||
|
||||
**Why this priority**: [Explain the value and why it has this priority level]
|
||||
|
||||
**Independent Test**: [Describe how this can be tested independently]
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** [initial state], **When** [action], **Then** [expected outcome]
|
||||
|
||||
---
|
||||
|
||||
### User Story 3 - [Brief Title] (Priority: P3)
|
||||
|
||||
[Describe this user journey in plain language]
|
||||
|
||||
**Why this priority**: [Explain the value and why it has this priority level]
|
||||
|
||||
**Independent Test**: [Describe how this can be tested independently]
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** [initial state], **When** [action], **Then** [expected outcome]
|
||||
|
||||
---
|
||||
|
||||
[Add more user stories as needed, each with an assigned priority]
|
||||
|
||||
### Edge Cases
|
||||
|
||||
<!--
|
||||
ACTION REQUIRED: The content in this section represents placeholders.
|
||||
Fill them out with the right edge cases.
|
||||
-->
|
||||
|
||||
- What happens when [boundary condition]?
|
||||
- How does system handle [error scenario]?
|
||||
|
||||
## Requirements *(mandatory)*
|
||||
|
||||
<!--
|
||||
ACTION REQUIRED: The content in this section represents placeholders.
|
||||
Fill them out with the right functional requirements.
|
||||
-->
|
||||
|
||||
### Functional Requirements
|
||||
|
||||
- **FR-001**: System MUST [specific capability, e.g., "allow users to create accounts"]
|
||||
- **FR-002**: System MUST [specific capability, e.g., "validate email addresses"]
|
||||
- **FR-003**: Users MUST be able to [key interaction, e.g., "reset their password"]
|
||||
- **FR-004**: System MUST [data requirement, e.g., "persist user preferences"]
|
||||
- **FR-005**: System MUST [behavior, e.g., "log all security events"]
|
||||
|
||||
*Example of marking unclear requirements:*
|
||||
|
||||
- **FR-006**: System MUST authenticate users via [NEEDS CLARIFICATION: auth method not specified - email/password, SSO, OAuth?]
|
||||
- **FR-007**: System MUST retain user data for [NEEDS CLARIFICATION: retention period not specified]
|
||||
|
||||
### Key Entities *(include if feature involves data)*
|
||||
|
||||
- **[Entity 1]**: [What it represents, key attributes without implementation]
|
||||
- **[Entity 2]**: [What it represents, relationships to other entities]
|
||||
|
||||
## Success Criteria *(mandatory)*
|
||||
|
||||
<!--
|
||||
ACTION REQUIRED: Define measurable success criteria.
|
||||
These must be technology-agnostic and measurable.
|
||||
-->
|
||||
|
||||
### Measurable Outcomes
|
||||
|
||||
- **SC-001**: [Measurable metric, e.g., "Users can complete account creation in under 2 minutes"]
|
||||
- **SC-002**: [Measurable metric, e.g., "System handles 1000 concurrent users without degradation"]
|
||||
- **SC-003**: [User satisfaction metric, e.g., "90% of users successfully complete primary task on first attempt"]
|
||||
- **SC-004**: [Business metric, e.g., "Reduce support tickets related to [X] by 50%"]
|
||||
251
nextflow/.specify/templates/tasks-template.md
Normal file
251
nextflow/.specify/templates/tasks-template.md
Normal file
@@ -0,0 +1,251 @@
|
||||
---
|
||||
|
||||
description: "Task list template for feature implementation"
|
||||
---
|
||||
|
||||
# Tasks: [FEATURE NAME]
|
||||
|
||||
**Input**: Design documents from `/specs/[###-feature-name]/`
|
||||
**Prerequisites**: plan.md (required), spec.md (required for user stories), research.md, data-model.md, contracts/
|
||||
|
||||
**Tests**: The examples below include test tasks. Tests are OPTIONAL - only include them if explicitly requested in the feature specification.
|
||||
|
||||
**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story.
|
||||
|
||||
## Format: `[ID] [P?] [Story] Description`
|
||||
|
||||
- **[P]**: Can run in parallel (different files, no dependencies)
|
||||
- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3)
|
||||
- Include exact file paths in descriptions
|
||||
|
||||
## Path Conventions
|
||||
|
||||
- **Single project**: `src/`, `tests/` at repository root
|
||||
- **Web app**: `backend/src/`, `frontend/src/`
|
||||
- **Mobile**: `api/src/`, `ios/src/` or `android/src/`
|
||||
- Paths shown below assume single project - adjust based on plan.md structure
|
||||
|
||||
<!--
|
||||
============================================================================
|
||||
IMPORTANT: The tasks below are SAMPLE TASKS for illustration purposes only.
|
||||
|
||||
The /speckit.tasks command MUST replace these with actual tasks based on:
|
||||
- User stories from spec.md (with their priorities P1, P2, P3...)
|
||||
- Feature requirements from plan.md
|
||||
- Entities from data-model.md
|
||||
- Endpoints from contracts/
|
||||
|
||||
Tasks MUST be organized by user story so each story can be:
|
||||
- Implemented independently
|
||||
- Tested independently
|
||||
- Delivered as an MVP increment
|
||||
|
||||
DO NOT keep these sample tasks in the generated tasks.md file.
|
||||
============================================================================
|
||||
-->
|
||||
|
||||
## Phase 1: Setup (Shared Infrastructure)
|
||||
|
||||
**Purpose**: Project initialization and basic structure
|
||||
|
||||
- [ ] T001 Create project structure per implementation plan
|
||||
- [ ] T002 Initialize [language] project with [framework] dependencies
|
||||
- [ ] T003 [P] Configure linting and formatting tools
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Foundational (Blocking Prerequisites)
|
||||
|
||||
**Purpose**: Core infrastructure that MUST be complete before ANY user story can be implemented
|
||||
|
||||
**⚠️ CRITICAL**: No user story work can begin until this phase is complete
|
||||
|
||||
Examples of foundational tasks (adjust based on your project):
|
||||
|
||||
- [ ] T004 Setup database schema and migrations framework
|
||||
- [ ] T005 [P] Implement authentication/authorization framework
|
||||
- [ ] T006 [P] Setup API routing and middleware structure
|
||||
- [ ] T007 Create base models/entities that all stories depend on
|
||||
- [ ] T008 Configure error handling and logging infrastructure
|
||||
- [ ] T009 Setup environment configuration management
|
||||
|
||||
**Checkpoint**: Foundation ready - user story implementation can now begin in parallel
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: User Story 1 - [Title] (Priority: P1) 🎯 MVP
|
||||
|
||||
**Goal**: [Brief description of what this story delivers]
|
||||
|
||||
**Independent Test**: [How to verify this story works on its own]
|
||||
|
||||
### Tests for User Story 1 (OPTIONAL - only if tests requested) ⚠️
|
||||
|
||||
> **NOTE: Write these tests FIRST, ensure they FAIL before implementation**
|
||||
|
||||
- [ ] T010 [P] [US1] Contract test for [endpoint] in tests/contract/test_[name].py
|
||||
- [ ] T011 [P] [US1] Integration test for [user journey] in tests/integration/test_[name].py
|
||||
|
||||
### Implementation for User Story 1
|
||||
|
||||
- [ ] T012 [P] [US1] Create [Entity1] model in src/models/[entity1].py
|
||||
- [ ] T013 [P] [US1] Create [Entity2] model in src/models/[entity2].py
|
||||
- [ ] T014 [US1] Implement [Service] in src/services/[service].py (depends on T012, T013)
|
||||
- [ ] T015 [US1] Implement [endpoint/feature] in src/[location]/[file].py
|
||||
- [ ] T016 [US1] Add validation and error handling
|
||||
- [ ] T017 [US1] Add logging for user story 1 operations
|
||||
|
||||
**Checkpoint**: At this point, User Story 1 should be fully functional and testable independently
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: User Story 2 - [Title] (Priority: P2)
|
||||
|
||||
**Goal**: [Brief description of what this story delivers]
|
||||
|
||||
**Independent Test**: [How to verify this story works on its own]
|
||||
|
||||
### Tests for User Story 2 (OPTIONAL - only if tests requested) ⚠️
|
||||
|
||||
- [ ] T018 [P] [US2] Contract test for [endpoint] in tests/contract/test_[name].py
|
||||
- [ ] T019 [P] [US2] Integration test for [user journey] in tests/integration/test_[name].py
|
||||
|
||||
### Implementation for User Story 2
|
||||
|
||||
- [ ] T020 [P] [US2] Create [Entity] model in src/models/[entity].py
|
||||
- [ ] T021 [US2] Implement [Service] in src/services/[service].py
|
||||
- [ ] T022 [US2] Implement [endpoint/feature] in src/[location]/[file].py
|
||||
- [ ] T023 [US2] Integrate with User Story 1 components (if needed)
|
||||
|
||||
**Checkpoint**: At this point, User Stories 1 AND 2 should both work independently
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: User Story 3 - [Title] (Priority: P3)
|
||||
|
||||
**Goal**: [Brief description of what this story delivers]
|
||||
|
||||
**Independent Test**: [How to verify this story works on its own]
|
||||
|
||||
### Tests for User Story 3 (OPTIONAL - only if tests requested) ⚠️
|
||||
|
||||
- [ ] T024 [P] [US3] Contract test for [endpoint] in tests/contract/test_[name].py
|
||||
- [ ] T025 [P] [US3] Integration test for [user journey] in tests/integration/test_[name].py
|
||||
|
||||
### Implementation for User Story 3
|
||||
|
||||
- [ ] T026 [P] [US3] Create [Entity] model in src/models/[entity].py
|
||||
- [ ] T027 [US3] Implement [Service] in src/services/[service].py
|
||||
- [ ] T028 [US3] Implement [endpoint/feature] in src/[location]/[file].py
|
||||
|
||||
**Checkpoint**: All user stories should now be independently functional
|
||||
|
||||
---
|
||||
|
||||
[Add more user story phases as needed, following the same pattern]
|
||||
|
||||
---
|
||||
|
||||
## Phase N: Polish & Cross-Cutting Concerns
|
||||
|
||||
**Purpose**: Improvements that affect multiple user stories
|
||||
|
||||
- [ ] TXXX [P] Documentation updates in docs/
|
||||
- [ ] TXXX Code cleanup and refactoring
|
||||
- [ ] TXXX Performance optimization across all stories
|
||||
- [ ] TXXX [P] Additional unit tests (if requested) in tests/unit/
|
||||
- [ ] TXXX Security hardening
|
||||
- [ ] TXXX Run quickstart.md validation
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Execution Order
|
||||
|
||||
### Phase Dependencies
|
||||
|
||||
- **Setup (Phase 1)**: No dependencies - can start immediately
|
||||
- **Foundational (Phase 2)**: Depends on Setup completion - BLOCKS all user stories
|
||||
- **User Stories (Phase 3+)**: All depend on Foundational phase completion
|
||||
- User stories can then proceed in parallel (if staffed)
|
||||
- Or sequentially in priority order (P1 → P2 → P3)
|
||||
- **Polish (Final Phase)**: Depends on all desired user stories being complete
|
||||
|
||||
### User Story Dependencies
|
||||
|
||||
- **User Story 1 (P1)**: Can start after Foundational (Phase 2) - No dependencies on other stories
|
||||
- **User Story 2 (P2)**: Can start after Foundational (Phase 2) - May integrate with US1 but should be independently testable
|
||||
- **User Story 3 (P3)**: Can start after Foundational (Phase 2) - May integrate with US1/US2 but should be independently testable
|
||||
|
||||
### Within Each User Story
|
||||
|
||||
- Tests (if included) MUST be written and FAIL before implementation
|
||||
- Models before services
|
||||
- Services before endpoints
|
||||
- Core implementation before integration
|
||||
- Story complete before moving to next priority
|
||||
|
||||
### Parallel Opportunities
|
||||
|
||||
- All Setup tasks marked [P] can run in parallel
|
||||
- All Foundational tasks marked [P] can run in parallel (within Phase 2)
|
||||
- Once Foundational phase completes, all user stories can start in parallel (if team capacity allows)
|
||||
- All tests for a user story marked [P] can run in parallel
|
||||
- Models within a story marked [P] can run in parallel
|
||||
- Different user stories can be worked on in parallel by different team members
|
||||
|
||||
---
|
||||
|
||||
## Parallel Example: User Story 1
|
||||
|
||||
```bash
|
||||
# Launch all tests for User Story 1 together (if tests requested):
|
||||
Task: "Contract test for [endpoint] in tests/contract/test_[name].py"
|
||||
Task: "Integration test for [user journey] in tests/integration/test_[name].py"
|
||||
|
||||
# Launch all models for User Story 1 together:
|
||||
Task: "Create [Entity1] model in src/models/[entity1].py"
|
||||
Task: "Create [Entity2] model in src/models/[entity2].py"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
### MVP First (User Story 1 Only)
|
||||
|
||||
1. Complete Phase 1: Setup
|
||||
2. Complete Phase 2: Foundational (CRITICAL - blocks all stories)
|
||||
3. Complete Phase 3: User Story 1
|
||||
4. **STOP and VALIDATE**: Test User Story 1 independently
|
||||
5. Deploy/demo if ready
|
||||
|
||||
### Incremental Delivery
|
||||
|
||||
1. Complete Setup + Foundational → Foundation ready
|
||||
2. Add User Story 1 → Test independently → Deploy/Demo (MVP!)
|
||||
3. Add User Story 2 → Test independently → Deploy/Demo
|
||||
4. Add User Story 3 → Test independently → Deploy/Demo
|
||||
5. Each story adds value without breaking previous stories
|
||||
|
||||
### Parallel Team Strategy
|
||||
|
||||
With multiple developers:
|
||||
|
||||
1. Team completes Setup + Foundational together
|
||||
2. Once Foundational is done:
|
||||
- Developer A: User Story 1
|
||||
- Developer B: User Story 2
|
||||
- Developer C: User Story 3
|
||||
3. Stories complete and integrate independently
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- [P] tasks = different files, no dependencies
|
||||
- [Story] label maps task to specific user story for traceability
|
||||
- Each user story should be independently completable and testable
|
||||
- Verify tests fail before implementing
|
||||
- Commit after each task or logical group
|
||||
- Stop at any checkpoint to validate story independently
|
||||
- Avoid: vague tasks, same file conflicts, cross-story dependencies that break independence
|
||||
24
nextflow/CITATION.cff
Normal file
24
nextflow/CITATION.cff
Normal file
@@ -0,0 +1,24 @@
|
||||
cff-version: 1.2.0
|
||||
message: "If you use this software, please cite it as below."
|
||||
authors:
|
||||
- family-names: "Di Tommaso"
|
||||
given-names: "Paolo"
|
||||
orcid: "https://orcid.org/0000-0003-3220-0253"
|
||||
- family-names: "Chatzou"
|
||||
given-names: "Maria"
|
||||
- family-names: "Floden"
|
||||
given-names: "Evan"
|
||||
orcid: "https://orcid.org/0000-0002-5431-190X"
|
||||
- family-names: "Prieto Barja"
|
||||
given-names: "Pablo"
|
||||
- family-names: "Palumbo"
|
||||
given-names: "Emilio"
|
||||
orcid: "https://orcid.org/0000-0003-4599-8161"
|
||||
- family-names: "Notredame"
|
||||
given-names: "Cedric"
|
||||
orcid: "https://orcid.org/0000-0003-1461-0988"
|
||||
title: "Nextflow enables reproducible computational workflows"
|
||||
version: 1.0.0
|
||||
doi: 10.1038/nbt.3820
|
||||
date-released: 2017-04-11
|
||||
url: "https://nextflow.io"
|
||||
141
nextflow/CLAUDE.md
Normal file
141
nextflow/CLAUDE.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Nextflow is a scientific workflow management system built primarily in Groovy and Java.
|
||||
It enables the creation of scalable, portable, and reproducible computational pipelines using a dataflow programming model.
|
||||
The project follows a modular architecture with a plugin-based system for cloud providers and specialized features.
|
||||
|
||||
## Development Commands
|
||||
|
||||
### Build System
|
||||
- **Primary build tool**: Gradle with wrapper (`./gradlew`)
|
||||
- **Quick commands via Makefile**:
|
||||
- `make compile` - Compile sources and export classpath
|
||||
- `make assemble` - Build info, compile, and assemble
|
||||
- `make test` - Run all tests
|
||||
- `make check` - Run verification tasks
|
||||
- `make clean` - Clean build artifacts
|
||||
|
||||
### Testing
|
||||
- **Unit tests**: `make test` or `./gradlew test`
|
||||
- Uses Spock Framework (Groovy-based testing)
|
||||
- JaCoCo integration for code coverage
|
||||
- **Specific test class**: `./gradlew test --tests "SomeTestClass"`
|
||||
- **Specific module**: `make test module=nextflow`
|
||||
- **Smoke tests**: `make smoke` or `NXF_SMOKE=1 ./gradlew test` (skips long-running tests, network-dependent tests, and cloud provider integration tests)
|
||||
- **Integration tests**: Run from `tests/` directory using test runner scripts
|
||||
- **Cloud validation tests**: Located in `validation/` directory (requires credentials)
|
||||
|
||||
### Development Workflow
|
||||
- **Development launcher**: `./launch.sh run script.nf` (uses development build)
|
||||
- **Dependency analysis**: `make deps` or `make deps config=runtime`
|
||||
- **Install locally**: `make install` (installs to Maven local)
|
||||
|
||||
## Architecture
|
||||
|
||||
### Core Modules (`modules/`)
|
||||
- **nextflow**: Main application module with core workflow engine, CLI, AST transformations, executors
|
||||
- **nf-commons**: Shared utilities, plugin system infrastructure, extension methods
|
||||
- **nf-httpfs**: HTTP filesystem support and custom providers
|
||||
- **nf-lang**: Language parsing, ANTLR grammars, AST implementation
|
||||
- **nf-lineage**: Data lineage tracking and workflow execution history
|
||||
|
||||
### Plugin System (`plugins/`)
|
||||
- **Cloud providers**: nf-amazon (AWS), nf-azure (Azure), nf-google (GCP)
|
||||
- **Execution platforms**: nf-k8s (Kubernetes)
|
||||
- **Services**: nf-tower (Seqera Platform), nf-wave (container management)
|
||||
- **Other**: nf-console (interactive interface), nf-cloudcache (cloud caching)
|
||||
|
||||
### Key Technologies
|
||||
- **Language**: Groovy 4.0.29 (Java-compatible, targeting Java 17)
|
||||
- **Concurrency**: GPars 1.2.1 (Actor model, parallel/concurrent programming)
|
||||
- **Build**: Gradle with Java 21 toolchain
|
||||
- **Parsing**: ANTLR for Nextflow DSL
|
||||
- **Serialization**: Kryo
|
||||
- **Database**: LevelDB for local caching
|
||||
- **Version Control**: JGit integration
|
||||
|
||||
### Testing Structure
|
||||
- **Unit tests**: Each module has `src/test/groovy/` with Spock Framework tests
|
||||
- **Integration tests**: `tests/` directory with .nf workflows and expected outputs
|
||||
- **Legacy tests**: `tests-v1/` for DSL v1 compatibility
|
||||
- **Validation tests**: `validation/` directory for cloud provider end-to-end testing
|
||||
- **Documentation tests**: `docs/snippets/` for verifying documentation examples
|
||||
|
||||
## Development Notes
|
||||
|
||||
### Code Standards
|
||||
- All code must include Apache 2.0 license headers
|
||||
- Contributions require Developer Certificate of Origin (DCO) sign-off
|
||||
- Use existing code patterns and conventions from similar modules
|
||||
- Follow Groovy idioms and leverage the Nextflow DSL patterns
|
||||
|
||||
### Common Development Tasks
|
||||
- **Local development**: Use `make compile` to build and `./launch.sh` to test changes
|
||||
- **Adding features**: First check modules like `nextflow` for core features or create plugins for specialized functionality
|
||||
- **Plugin development**: Follow existing plugin patterns in `plugins/` directory
|
||||
- **Testing changes**: Always run `make test` before committing
|
||||
- **Cloud testing**: Use validation scripts in `validation/` directory with appropriate credentials
|
||||
|
||||
### Build Configuration
|
||||
- Java toolchain uses version 21 for development, targets Java 17 compatibility
|
||||
- Uses shadow plugin for creating fat JARs
|
||||
- Maven publication to S3-based Seqera repositories
|
||||
- Multi-module project with shared dependencies managed in root build.gradle
|
||||
|
||||
### Git conventions
|
||||
|
||||
- **DCO sign-off required**: All commits must be signed by adding a `Signed-off-by` line to the commit message or by using the `-s` option (see CONTRIBUTING.md for details).
|
||||
- **Always use sign-off**: Use `git commit -s` or `git commit --signoff` for commits to avoid DCO bot issues
|
||||
- **CI control tags**: Use special tags in commit messages to control CI behavior:
|
||||
- `[ci skip]` - Skip the execution of CI tests
|
||||
- `[ci fast]` - Run only unit tests and skip integration tests
|
||||
- `[e2e stage]` - Run end-to-end tests vs Seqera platform stage environment
|
||||
- `[e2e prod]` - Same but against production platform
|
||||
- `[release]` - Trigger release process
|
||||
|
||||
## Important Files
|
||||
- `VERSION`: Define the current version number
|
||||
- `nextflow`: Launch wrapper script (updated by build process)
|
||||
- `.launch.classpath`: Development classpath (generated by `make compile`)
|
||||
- `build.gradle`: Root build configuration with multi-module setup
|
||||
- `settings.gradle`: Gradle project structure definition
|
||||
- `plugins/*/VERSION`: Define the version of the corresponding plugin sub-project.
|
||||
- `adr/`: Architecture Decision Records (ADRs) documenting significant structural and technical decisions in the project
|
||||
|
||||
## Release process
|
||||
|
||||
Follow these actions to make a new release:
|
||||
|
||||
- Update the `changelog.txt` file in each plugin sub-project (if any change has been done).
|
||||
- Update the `VERSION` file in in each plugin sub-project.
|
||||
Use a semantic version number depending the impact of the change, or do not change
|
||||
if no changes have been done to the plugin.
|
||||
- Update `nextflowVersion` attribute in the `build.gradle` file for plugins requiring specific
|
||||
Nextflow versions.
|
||||
- Commit the version and changelog files changes independently for each plugin. Use as commit
|
||||
message the template `Bump plugin-name@version` e.g. `Bump nf-amazon@2.0.0.
|
||||
- Update `VERSION` file in the project root using a calendar-like versioning scheme. Versions in the 4-th and 10-th month are "stable releases", e.g. `25.10.0`, while versions in all other months are "edge releases", e.g. `25.09.0-edge`.
|
||||
- Update the project root `changelog.txt` with changes since the past release. Use the git log
|
||||
command to determine what changed e.g. `git log v<PREVIOUS VERSION>..`
|
||||
- Run `make releaseInfo` to update the version number and generate checksums.
|
||||
- Run this command to stage for commit the release files:
|
||||
```
|
||||
git add \
|
||||
VERSION \
|
||||
changelog.txt \
|
||||
nextflow \
|
||||
nextflow.md5 \
|
||||
nextflow.sha1 \
|
||||
nextflow.sha256 \
|
||||
modules/nextflow/src/main/resources/META-INF/plugins-info.txt \
|
||||
modules/nextflow/src/main/resources/META-INF/build-info.properties
|
||||
```
|
||||
- Make a commit using the `[release]` and `[e2e prod]` tags in the comment and push it upstream to trigger the release automation with GitHub action:
|
||||
```
|
||||
git commit -m "[release] Nextflow version 25.09.0-edge"
|
||||
git push origin master
|
||||
```
|
||||
77
nextflow/CODE-OF-CONDUCT.md
Normal file
77
nextflow/CODE-OF-CONDUCT.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
In the interest of fostering an open and welcoming environment, we as
|
||||
contributors and maintainers pledge to making participation in our project and
|
||||
our community a harassment-free experience for everyone, regardless of age, body
|
||||
size, disability, ethnicity, sex characteristics, gender identity and expression,
|
||||
level of experience, education, socioeconomic status, nationality, personal
|
||||
appearance, race, religion, or sexual identity and orientation.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to creating a positive environment
|
||||
include:
|
||||
|
||||
* Using welcoming and inclusive language
|
||||
* Being respectful of differing viewpoints and experiences
|
||||
* Gracefully accepting constructive criticism
|
||||
* Focusing on what is best for the community
|
||||
* Showing empathy towards other community members
|
||||
|
||||
Examples of unacceptable behavior by participants include:
|
||||
|
||||
* The use of sexualized language or imagery and unwelcome sexual attention or
|
||||
advances
|
||||
* Trolling, insulting/derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or electronic
|
||||
address, without explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Our Responsibilities
|
||||
|
||||
Project maintainers are responsible for clarifying the standards of acceptable
|
||||
behavior and are expected to take appropriate and fair corrective action in
|
||||
response to any instances of unacceptable behavior.
|
||||
|
||||
Project maintainers have the right and responsibility to remove, edit, or
|
||||
reject comments, commits, code, wiki edits, issues, and other contributions
|
||||
that are not aligned to this Code of Conduct, or to ban temporarily or
|
||||
permanently any contributor for other behaviors that they deem inappropriate,
|
||||
threatening, offensive, or harmful.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies within all project spaces, and it also applies when
|
||||
an individual is representing the project or its community in public spaces.
|
||||
Examples of representing a project or community include using an official
|
||||
project e-mail address, posting via an official social media account, or acting
|
||||
as an appointed representative at an online or offline event. Representation of
|
||||
a project may be further defined and clarified by project maintainers.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported by contacting the project team at info@nextflow.io. All
|
||||
complaints will be reviewed and investigated and will result in a response that
|
||||
is deemed necessary and appropriate to the circumstances. The project team is
|
||||
obligated to maintain confidentiality with regard to the reporter of an incident.
|
||||
Further details of specific enforcement policies may be posted separately.
|
||||
|
||||
Project maintainers who do not follow or enforce the Code of Conduct in good
|
||||
faith may face temporary or permanent repercussions as determined by other
|
||||
members of the project's leadership.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
|
||||
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
|
||||
For answers to common questions about this code of conduct, see
|
||||
https://www.contributor-covenant.org/faq
|
||||
|
||||
47
nextflow/CONTRIBUTING.md
Normal file
47
nextflow/CONTRIBUTING.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# Contributing to Nextflow
|
||||
|
||||
This guide documents the various ways to contribute to Nextflow, including what is required before submitting a code change.
|
||||
|
||||
Contributing to Nextflow doesn't just mean writing code. Helping new users in the community, testing releases and bug fixes, and improving documentation are all essential and valuable contributions. Helping in these ways is an excellent way to become an effective contributor and gain credibility within the community, which makes it easier to make larger contributions like code changes and new features.
|
||||
|
||||
## Helping Other Users
|
||||
|
||||
A great way to contribute to Nextflow is to answer user questions on the [community forum](https://community.seqera.io) and the [Nextflow Slack](https://www.nextflow.io/slack-invite.html). Contributors should ideally be active members here and keep up with the latest developments in the Nextflow community. There are always many new Nextflow users, so taking a few minutes to help answer a question is a valuable community service and a great way to demonstrate your expertise.
|
||||
|
||||
## Documentation Changes
|
||||
|
||||
Propose changes to the [Nextflow documentation](https://nextflow.io/docs/latest/) by editing the source files in the [docs](https://github.com/nextflow-io/nextflow/tree/master/docs) directory. The `README.md` in that directory describes how to build and preview the docs locally. Finally, open a pull request with the proposed changes.
|
||||
|
||||
## Bug Reports
|
||||
|
||||
Submitting a bug report is one of the simplest and most useful ways to contribute, as it helps us to quickly identify and fix issues and thereby make Nextflow more stable.
|
||||
|
||||
Report a bug using the **New issue** button on the [issues page](https://github.com/nextflow-io/nextflow/issues). A good bug report should include a minimal test case that can replicate the reported bug. Please follow the instructions in the issue template when submitting a bug report.
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
Contributing bug fixes is the best way to gain experience with the Nextflow codebase and credibility within the community as a project contributor.
|
||||
|
||||
If you are new to the Nextflow codebase and want to get involved, check out issues marked as [`help wanted`](https://github.com/nextflow-io/nextflow/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) or [`good first issue`](https://github.com/nextflow-io/nextflow/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22+). Feel free to ask for help if you get stuck while trying to implement a fix!
|
||||
|
||||
## New Features
|
||||
|
||||
Before contributing a new feature, please submit a new feature proposal on the [issues page](https://github.com/nextflow-io/nextflow/issues) and discuss it with the community.
|
||||
|
||||
Submitting a proposal helps identify possible overlaps with other planned features and avoid potential misunderstandings, conflicts, and wasted effort.
|
||||
|
||||
## Code Changes
|
||||
|
||||
When submitting a contribution, you will be required to sign a [Developer Certificate of Origin (DCO)](https://developercertificate.org/) to certify that you are the author of the source code or otherwise have the right to submit it to the project.
|
||||
|
||||
Contributor signatures are provided by adding a `Signed-off-by` line to the commit message as shown below, or by using the `-s` option with [`git commit`](https://help.github.com/articles/signing-commits/). For example:
|
||||
|
||||
```
|
||||
This is my commit message
|
||||
|
||||
Signed-off-by: Random J Developer <random@developer.example.org>
|
||||
```
|
||||
|
||||
The process is automatically managed by the [Probot](https://probot.github.io/apps/dco/) app for GitHub.
|
||||
|
||||
For more information about working on the Nextflow source code, visit the [Nextflow docs](https://nextflow.io/docs/latest/developer/).
|
||||
176
nextflow/COPYING
Normal file
176
nextflow/COPYING
Normal file
@@ -0,0 +1,176 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
106
nextflow/GIT-README.md
Normal file
106
nextflow/GIT-README.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# GIT README
|
||||
|
||||
Shortcuts to recurrent Git commands
|
||||
|
||||
## Sync remote fork
|
||||
|
||||
```bash
|
||||
git fetch public
|
||||
git checkout master
|
||||
git merge public/master
|
||||
```
|
||||
|
||||
List branch remote
|
||||
|
||||
git branch -vv
|
||||
|
||||
Checkout from a remote branch
|
||||
|
||||
git co -b <local name> upstream/master
|
||||
|
||||
Push to a remote upstream branch
|
||||
|
||||
git push <remote> <local branch>:<remote branch>
|
||||
|
||||
eg:
|
||||
|
||||
git push upstream foo:master
|
||||
|
||||
Read more [here](https://help.github.com/articles/syncing-a-fork/).
|
||||
|
||||
## Pull options
|
||||
|
||||
git config pull.rebase false # merge (the default strategy)
|
||||
git config pull.rebase true # rebase
|
||||
git config pull.ff only # fast-forward only
|
||||
|
||||
|
||||
## Subtree
|
||||
|
||||
The `tests` directory is a Git subtree created with the
|
||||
following commands:
|
||||
|
||||
git remote add tests git@github.com:nextflow-io/tests.git
|
||||
git subtree add --squash --prefix=tests/ tests integration
|
||||
|
||||
|
||||
To pull changes from the [tests repo](https://github.com/nextflow-io/tests) use this command:
|
||||
|
||||
git subtree pull --squash --prefix=tests/ tests integration
|
||||
|
||||
To push changes to the [tests repo](https://github.com/nextflow-io/tests) use this command:
|
||||
|
||||
git subtree push --prefix=tests/ tests integration
|
||||
|
||||
|
||||
Read more [here](https://andrey.nering.com.br/2016/git-submodules-vs-subtrees/).
|
||||
|
||||
## Stash shortcuts
|
||||
|
||||
git stash list
|
||||
git stash pop
|
||||
git stash pop stash@{1}
|
||||
git showtool stash@{0}
|
||||
git stash drop
|
||||
git stash drop stash@{1}
|
||||
git stash clear
|
||||
git diff stash
|
||||
git diff stash@{1} [other]
|
||||
|
||||
## Misc
|
||||
|
||||
Find a commit in any branch introducing a change
|
||||
|
||||
git log -S <whatever> --source --all
|
||||
|
||||
Reset last merge pushed
|
||||
|
||||
git reset --hard HEAD@{1}
|
||||
|
||||
Read more https://stackoverflow.com/a/11722640/395921
|
||||
|
||||
## GPG keys
|
||||
|
||||
To sign Git commits with a GPG key on Mac use [GPG Suite](https://gpgtools.org/), import your key, then:
|
||||
|
||||
git config --global gpg.program /usr/local/MacGPG2/bin/gpg2
|
||||
git config --global user.signingkey <your key>
|
||||
git config --global commit.gpgsign true
|
||||
git config --global format.signoff true ## TO AVOID TO SPECIFY -S option each time
|
||||
|
||||
Read more:
|
||||
https://gist.github.com/danieleggert/b029d44d4a54b328c0bac65d46ba4c65
|
||||
|
||||
|
||||
## Change Git history root
|
||||
|
||||
https://stackoverflow.com/questions/4515580/how-do-i-remove-the-old-history-from-a-git-repository
|
||||
|
||||
# Pull master while in a another branch
|
||||
|
||||
git fetch origin master:master
|
||||
|
||||
# Alternative to `git pull`
|
||||
|
||||
Use `git fetch && git rebase`
|
||||
|
||||
142
nextflow/Makefile
Normal file
142
nextflow/Makefile
Normal file
@@ -0,0 +1,142 @@
|
||||
#
|
||||
# Copyright 2013-2024, Seqera Labs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
|
||||
#
|
||||
# add parameters to the command line as `param=value` for example:
|
||||
# make deps config=runtime
|
||||
#
|
||||
|
||||
config ?= compileClasspath
|
||||
|
||||
ifdef module
|
||||
mm = :${module}:
|
||||
else
|
||||
mm =
|
||||
endif
|
||||
|
||||
compile:
|
||||
./gradlew compile exportClasspath
|
||||
@echo "DONE `date`"
|
||||
|
||||
assemble:
|
||||
./gradlew buildInfo compile assemble
|
||||
|
||||
releaseInfo:
|
||||
./gradlew releaseInfo
|
||||
|
||||
check:
|
||||
./gradlew check
|
||||
|
||||
clean:
|
||||
rm -rf .nextflow*
|
||||
rm -rf work
|
||||
rm -rf modules/nextflow/.nextflow*
|
||||
rm -rf modules/nextflow/work
|
||||
rm -rf build
|
||||
rm -rf modules/*/build
|
||||
rm -rf plugins/*/build
|
||||
./gradlew clean
|
||||
|
||||
#
|
||||
# install compiled artifacts in Maven local dir
|
||||
#
|
||||
install:
|
||||
BUILD_PACK=1 \
|
||||
./gradlew installLauncher publishToMavenLocal installPlugin
|
||||
|
||||
installScratch:
|
||||
BUILD_PACK=1 \
|
||||
./gradlew installScratch publishToMavenLocal installPlugin
|
||||
|
||||
#
|
||||
# Show dependencies try `make deps config=runtime`, `make deps config=google`
|
||||
#
|
||||
deps:
|
||||
./gradlew -q ${mm}dependencies --configuration ${config}
|
||||
|
||||
deps-all:
|
||||
./gradlew -q dependencyInsight --configuration ${config} --dependency ${module}
|
||||
|
||||
#
|
||||
# Refresh SNAPSHOTs dependencies
|
||||
#
|
||||
refresh:
|
||||
./gradlew --refresh-dependencies
|
||||
|
||||
#
|
||||
# Run all tests or selected ones
|
||||
#
|
||||
test:
|
||||
ifndef class
|
||||
./gradlew ${mm}test
|
||||
else
|
||||
./gradlew ${mm}test --tests ${class}
|
||||
endif
|
||||
|
||||
#
|
||||
# Run smoke tests
|
||||
#
|
||||
smoke:
|
||||
NXF_SMOKE=1 ./gradlew ${mm}test
|
||||
|
||||
#
|
||||
# Upload JAR artifacts to Maven Central
|
||||
#
|
||||
upload:
|
||||
./gradlew upload
|
||||
|
||||
#
|
||||
# Create self-contained distribution package
|
||||
#
|
||||
pack:
|
||||
BUILD_PACK=1 ./gradlew pack
|
||||
|
||||
#
|
||||
# Upload NF launcher to nextflow.io web site
|
||||
#
|
||||
deploy:
|
||||
BUILD_PACK=1 ./gradlew deploy
|
||||
|
||||
#
|
||||
# Close artifacts uploaded to Maven central
|
||||
#
|
||||
close:
|
||||
./gradlew closeAndReleaseRepository
|
||||
|
||||
#
|
||||
# Upload final package to GitHub
|
||||
#
|
||||
release:
|
||||
BUILD_PACK=1 ./gradlew release
|
||||
|
||||
#
|
||||
# Create and upload docker image distribution
|
||||
#
|
||||
dockerImage:
|
||||
BUILD_PACK=1 ./gradlew dockerImage
|
||||
|
||||
#
|
||||
# Create local docker image
|
||||
#
|
||||
dockerPack:
|
||||
BUILD_PACK=1 ./gradlew publishToMavenLocal dockerPack -Dmaven.repo.local=${PWD}/build/docker/.nextflow/capsule/deps/ installPlugin
|
||||
|
||||
release-plugins:
|
||||
./gradlew releasePluginToRegistryIfNotExists
|
||||
|
||||
publish-artifacts:
|
||||
./gradlew publishAllPublicationsToSeqeraRepository
|
||||
|
||||
149
nextflow/NOTICE
Normal file
149
nextflow/NOTICE
Normal file
@@ -0,0 +1,149 @@
|
||||
NEXTFLOW
|
||||
|
||||
Copyright 2013-2024, Seqera Labs
|
||||
|
||||
|
||||
This software includes source code and libraries developed by:
|
||||
|
||||
Lorenz Gerber
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
SciLifeLab
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Synthetic Genomics
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
University of Tübingen, Quantitative Biology Center (QBiC)
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
WuxiNextcode
|
||||
Copyright 2018, WuxiNextcode
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Apache Commons Lang
|
||||
Copyright Apache Software Foundation
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Apache Commons Codec
|
||||
Copyright Apache Software Foundation
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Apache Http Components
|
||||
Copyright Apache Software Foundation
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Apache Groovy
|
||||
Copyright 2003-2018 Apache Software Foundation
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
AWS SDK for Java
|
||||
Copyright 2010-2014 Amazon.com, Inc. or its affiliates
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Azure Cloud plugin for Nextflow
|
||||
Copyright 2021, Microsoft Corp
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Azure SDK for Java
|
||||
Copyright (c) 2015 Microsoft Corp
|
||||
Licensed under MIT License (MIT)
|
||||
|
||||
Failsafe
|
||||
Copyright Jonathan Halterman and friends
|
||||
Released under the Apache 2.0 license
|
||||
|
||||
Gpars
|
||||
Copyright (c) 2008-2011, The Gpars authors
|
||||
Licensed under the Apache License, Version 2.0
|
||||
|
||||
Grengine
|
||||
https://gitlab.com/jexler/grengine
|
||||
Copyright (c) 2014-now, Alain Stalder
|
||||
Licensed under the Apache License, Version 2.0
|
||||
|
||||
Google plugin for Nextflow
|
||||
Copyright 2019, Google Inc
|
||||
Licensed under the Apache License, Version 2.0
|
||||
|
||||
Google API Java client
|
||||
Copyright 2019, Google Inc
|
||||
Licensed under the Apache License, Version 2.0
|
||||
|
||||
Gson
|
||||
https://github.com/google/gson
|
||||
Copyright 2008 Google Inc.
|
||||
Licensed under the Apache License, Version 2.0
|
||||
|
||||
Guava
|
||||
https://github.com/google/guava
|
||||
Copyright 2008 The Guava Authors
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
JGit
|
||||
Copyright JGit authors
|
||||
Licensed under the Eclipse Distribution License - v 1.0
|
||||
|
||||
Javaewah
|
||||
https://github.com/lemire/javaewah
|
||||
Copyright 2009-2016, Daniel Lemire, Cliff Moon, David McIntosh, Robert Becho, Google Inc., Veronika Zenz, Owen Kaser, Gregory Ssi-Yan-Kai, Rory Graves
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Jackson
|
||||
https://github.com/FasterXML/jackson-core
|
||||
https://github.com/FasterXML/jackson-annotations
|
||||
https://github.com/FasterXML/jackson-databind
|
||||
Copyright The Jackson authors
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
JCommander
|
||||
Copyright (c) 2010 Cedric Beust
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Jsoup
|
||||
https://github.com/jhy/jsoup
|
||||
Copyright 2009-2018 Jonathan Hedley
|
||||
Licensed under the MIT license
|
||||
|
||||
Kryo
|
||||
Copyright (c) 2008-2018, Nathan Sweet
|
||||
Licensed under BSD 3-Clause license
|
||||
|
||||
Kryo serializers
|
||||
https://github.com/magro/kryo-serializers
|
||||
Copyright 2010 Martin Grotzke
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Leveldb, https://github.com/dain/leveldb
|
||||
Copyright 2011 Dain Sundstrom dain@iq80.com
|
||||
Copyright 2011 FuseSource Corp. http://fusesource.com
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Logback
|
||||
Copyright 1999-2017, QOS.ch
|
||||
Licensed under the EPL v1.0 license
|
||||
|
||||
Objenesis
|
||||
https://github.com/easymock/objenesis
|
||||
Copyright 2006-2018 the original author or authors
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Okhttp
|
||||
Copyright 2013-2016 Square, Inc.
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
SLF4J
|
||||
Copyright 2004-2018 QOS.ch
|
||||
Licensed under the MIT license
|
||||
|
||||
Snakeyaml
|
||||
Copyright 2008, http://www.snakeyaml.org
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Spack
|
||||
https://github.com/spack/spack
|
||||
Copyright 2013-2024 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
Licensed under Apache License, Version 2.0
|
||||
|
||||
Nextflow is a registered trademark of Seqera Labs, Spain.
|
||||
See the trademark policy for details https://github.com/nextflow-io/trademark
|
||||
78
nextflow/README.md
Normal file
78
nextflow/README.md
Normal file
@@ -0,0 +1,78 @@
|
||||
<h1 align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="docs/_static/nextflow-logo-bg-dark.png">
|
||||
<source media="(prefers-color-scheme: light)" srcset="docs/_static/nextflow-logo-bg-light.png">
|
||||
<img width="500" alt="Nextflow" src="docs/_static/nextflow-logo-bg-light.png">
|
||||
</picture>
|
||||
</h1>
|
||||
|
||||
_"Dataflow variables are spectacularly expressive in concurrent programming"_
|
||||
<br>[Henri E. Bal , Jennifer G. Steiner , Andrew S. Tanenbaum](https://dl.acm.org/doi/abs/10.1145/72551.72552)
|
||||
|
||||
[](https://github.com/nextflow-io/nextflow/actions/workflows/build.yml?query=branch%3Amaster+event%3Apush)
|
||||
[](https://github.com/nextflow-io/nextflow/releases/latest)
|
||||
[](https://twitter.com/nextflowio)
|
||||
[](https://www.nature.com/articles/nbt.3820)
|
||||
[](http://bioconda.github.io/recipes/nextflow/README.html)
|
||||
[](https://github.com/nextflow-io/nextflow/blob/master/COPYING)
|
||||
|
||||
Nextflow is a workflow system for creating scalable, portable, and reproducible workflows. It is based on the dataflow programming model, which greatly simplifies the writing of parallel and distributed pipelines, allowing you to focus on the flow of data and computation. Nextflow can deploy workflows on a variety of execution platforms, including your local machine, HPC schedulers, AWS Batch, Azure Batch, Google Cloud Batch, and Kubernetes. Additionally, it supports many ways to manage your software dependencies, including Conda, Spack, Docker, Podman, Singularity, and more.
|
||||
|
||||
Nextflow is built with ❤️ by [Seqera](https://seqera.io/), the best place to run Nextflow.
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="docs/_static/seqera-logo-light.svg">
|
||||
<source media="(prefers-color-scheme: light)" srcset="docs/_static/seqera-logo-dark.svg">
|
||||
<img width="200" alt="Seqera" src="docs/_static/seqera-logo-dark.svg">
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
## Quick start
|
||||
|
||||
Install Nextflow with a single command:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://get.nextflow.io | bash
|
||||
```
|
||||
|
||||
It creates the `nextflow` executable file in the current directory. You can then move it to a directory in your `$PATH` to run it from anywhere.
|
||||
|
||||
Nextflow can also be installed from Bioconda:
|
||||
|
||||
```bash
|
||||
conda install -c bioconda nextflow
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
The Nextflow documentation is available for the latest [stable](https://nextflow.io/docs/latest/) and [edge](https://nextflow.io/docs/edge/) releases.
|
||||
|
||||
## Community
|
||||
|
||||
You can post questions and get help in the [Nextflow community forum](https://community.seqera.io) or the [Nextflow Slack](https://www.nextflow.io/slack-invite.html). Bugs and feature requests should be reported as [GitHub issues](https://github.com/nextflow-io/nextflow/issues/new/choose).
|
||||
|
||||
The Nextflow community is highly active with regular community meetings, events, a podcast and more. You can view much of this material on the [Nextflow](https://www.youtube.com/@Nextflow) and [nf-core](https://www.youtube.com/@nf-core) YouTube channels.
|
||||
|
||||
The [nf-core](https://nf-co.re/) project is a community effort aggregating high quality Nextflow workflows which can be used by everyone.
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are more than welcome. See the [CONTRIBUTING](CONTRIBUTING.md) file for details.
|
||||
|
||||
## License
|
||||
|
||||
Nextflow is released under the Apache 2.0 license. Nextflow is a [registered trademark](https://github.com/nextflow-io/trademark).
|
||||
|
||||
## Citations
|
||||
|
||||
If you use Nextflow in your work, please cite:
|
||||
|
||||
P. Di Tommaso, et al. Nextflow enables reproducible computational workflows. Nature Biotechnology 35, 316–319 (2017) doi:[10.1038/nbt.3820](http://www.nature.com/nbt/journal/v35/n4/full/nbt.3820.html)
|
||||
|
||||
## Credits
|
||||
|
||||
Nextflow is built on two \*great\* open-source software projects, <a href='http://groovy-lang.org' target='_blank'>Groovy</a>
|
||||
and <a href='http://www.gpars.org/' target='_blank'>GPars</a>.
|
||||
|
||||
<a href='http://www.yourkit.com' target='_blank'>YourKit</a> is kindly supporting Nextflow with its fully-featured Java Profiler.
|
||||
1
nextflow/VERSION
Normal file
1
nextflow/VERSION
Normal file
@@ -0,0 +1 @@
|
||||
26.04.0
|
||||
229
nextflow/adr/20250825-workflow-params.md
Normal file
229
nextflow/adr/20250825-workflow-params.md
Normal file
@@ -0,0 +1,229 @@
|
||||
# Workflow params
|
||||
|
||||
- Authors: Ben Sherman
|
||||
- Status: accepted
|
||||
- Date: 2025-08-25
|
||||
- Tags: lang, static-types, params
|
||||
|
||||
## Summary
|
||||
|
||||
Introduce a unified, statically typed way to declare the top-level inputs (i.e. parameters) of a workflow.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Pipeline parameters in Nextflow are currently declared using property assignments:
|
||||
|
||||
```groovy
|
||||
params.reads = "$baseDir/data/ggal/ggal_gut_{1,2}.fq"
|
||||
params.transcriptome = "$baseDir/data/ggal/ggal_1_48850000_49020000.Ggal71.500bpflank.fa"
|
||||
params.multiqc = "$baseDir/multiqc"
|
||||
```
|
||||
|
||||
This approach has several limitations:
|
||||
|
||||
- **No type annotations**: Parameter types cannot be expressed in the script. The type of a parameter can only be inferred from its default value, which may be ambiguous (e.g., a default value of `null`, a `String` that should be treated as a `Path`).
|
||||
|
||||
- **Heuristic type coercion**: When a parameter is supplied on the command line, Nextflow attempts to coerce the string value to the appropriate type using heuristics (e.g., `'true'` → boolean `true`, `'42'` → integer `42`). These heuristics are not always correct and can lead to unexpected behavior.
|
||||
|
||||
- **No built-in validation**: There is no built-in way to validate that a parameter is required, or that a parameter value has the correct type. Validation must be done manually in the script, or through an external JSON Schema file (`nextflow_schema.json`).
|
||||
|
||||
- **Scattered declarations and usage**: Parameters may be declared anywhere in the script or across multiple scripts, making it difficult to get a single view of the pipeline parameters. Parameters can be used anywhere in the pipeline, even outside the script where they are declared, making it impossible to validate params usage at compile-time.
|
||||
|
||||
## Goals
|
||||
|
||||
- Declare all parameters in one place in the script, with documentation.
|
||||
|
||||
- Provide explicit type annotations for parameters, enabling compile-time validation and IDE support.
|
||||
|
||||
- Clearly distinguish between required and optional parameters.
|
||||
|
||||
- Coerce CLI parameter values based on declared types, rather than relying on heuristics.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Removing the legacy `params.foo = bar` syntax -- legacy parameters must continue to work without modification.
|
||||
|
||||
- Changing the `params` config scope -- params can still be declared in the config file, although some best practices apply.
|
||||
|
||||
- Replacing `nextflow_schema.json` -- while the `params` block addresses many of the same needs, existing pipelines that use a JSON Schema should not be required to migrate. A native integration with `nextflow_schema.json` can be explored in the future.
|
||||
|
||||
- Supporting nested params -- the `params` block only supports a flat list of params. Nested params can still be used in the config, but they do not have first-class support at this time.
|
||||
|
||||
## Decision
|
||||
|
||||
Introduce the `params` block for declaring pipeline parameters. Each parameter is declared with a name, a type, and an optional default value:
|
||||
|
||||
```groovy
|
||||
params {
|
||||
// Path to the input samplesheet
|
||||
input: Path
|
||||
|
||||
// Whether to save intermediate files
|
||||
save_intermeds: Boolean = false
|
||||
}
|
||||
```
|
||||
|
||||
Typed parameters are used to validate parameter usage in the script, and to coerce CLI parameter values at runtime.
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
### Parameter declaration
|
||||
|
||||
The `params` block consists of parameter *declarations*. Each parameter is declared as `name: Type` (required) or `name: Type = default` (optional with default):
|
||||
|
||||
```groovy
|
||||
params {
|
||||
input: Path // required
|
||||
extra_file: Path? // optional (defaults to null)
|
||||
db_file: Path = 'db.json' // optional with default
|
||||
flag: Boolean // boolean params default to false
|
||||
}
|
||||
```
|
||||
|
||||
All standard Nextflow types except `Channel` and `Value` can be used for parameter type annotations.
|
||||
|
||||
### Required and optional parameters
|
||||
|
||||
A parameter without a default value is *required*. If a required parameter is not supplied at runtime (via the command line, a params file, or the config), the run fails immediately with an informative error.
|
||||
|
||||
A parameter with the `?` suffix on its type is *optional* and will be `null` if not supplied. Boolean parameters without a default value implicitly default to `false`.
|
||||
|
||||
### Type-based CLI coercion
|
||||
|
||||
When a parameter is supplied on the command line, Nextflow converts the string value to the declared type:
|
||||
|
||||
| Declared type | String input | Resolved value |
|
||||
|---|---|---|
|
||||
| `Boolean` | `'true'` | `true` |
|
||||
| `Integer` | `'42'` | `42` |
|
||||
| `Float` | `'3.14'` | `3.14` |
|
||||
| `Duration` | `'1h'` | `Duration.of('1h')` |
|
||||
| `MemoryUnit` | `'8 GB'` | `MemoryUnit.of('8 GB')` |
|
||||
| `Path` | `'/data'` | `Path.of('/data')` |
|
||||
|
||||
This replaces the heuristic type detection used for legacy parameters.
|
||||
|
||||
### Compile-time validation
|
||||
|
||||
Legacy parameters can be accessed globally by all scripts in the pipeline. While this approach is flexible, it prevents compile-time validation and breaks modularity.
|
||||
|
||||
When a module references a param, it implicitly assumes that the param will always be defined by the workflow that uses it. This assumption cannot be validated at compile-time, so if the param is missing, an error will occur only at runtime.
|
||||
|
||||
The `params` block solves this problem by defining all params in one place. It serves as the inputs for the entry workflow, similar to the `take:` section in named workflows. Parameters should be passed to processes and workflows as explicit inputs, so that every variable reference can be validated against local declarations.
|
||||
|
||||
For example, the following workflow:
|
||||
|
||||
```groovy
|
||||
// main.nf
|
||||
params.input = '...'
|
||||
|
||||
workflow {
|
||||
HELLO()
|
||||
}
|
||||
|
||||
// hello.nf
|
||||
workflow HELLO {
|
||||
println "input = ${params.input}"
|
||||
}
|
||||
```
|
||||
|
||||
Can be rewritten as follows:
|
||||
|
||||
```groovy
|
||||
// main.nf
|
||||
params {
|
||||
input: String
|
||||
}
|
||||
|
||||
workflow {
|
||||
HELLO(params.input)
|
||||
}
|
||||
|
||||
// hello.nf
|
||||
workflow HELLO {
|
||||
take:
|
||||
input: String
|
||||
|
||||
main:
|
||||
println "input = ${input}"
|
||||
}
|
||||
```
|
||||
|
||||
Typed parameters can still be used globally by all scripts for backwards compatibility. However, the type checker will only validate params used in the entry workflow and `output` block. Users should eventually migrate their pipelines as shown above for effective type checking.
|
||||
|
||||
### Script and config params
|
||||
|
||||
Parameters can also be defined in config files:
|
||||
|
||||
```groovy
|
||||
params {
|
||||
outdir = 'results'
|
||||
publish_dir_mode = 'copy'
|
||||
}
|
||||
```
|
||||
|
||||
Config params continue to work as before. As a best practice, they should be used only to "configure the configuration."
|
||||
|
||||
Some config params can be replaced with native functionality, e.g., `outputDir` and `workflow.output.mode` for the above. The nf-core [institutional configs](https://github.com/nf-core/configs), which enable users to run a pipeline with their institutional config entirely from the command line, cannot be easily replaced and provide a clear use case for config params.
|
||||
|
||||
Config params are also propagated to the script since the config file can overwrite script params (e.g. in a profile). However, since the script `params` block only allows params that were explicitly declared, it needs to be able to distinguish between config params and invalid params (e.g. command line param with a typo).
|
||||
|
||||
To prevent a circular dependency between the script execution and config resolution, parameters are resolved as follows:
|
||||
|
||||
1. Load *CLI params* from command line, params file
|
||||
|
||||
2. Load config files
|
||||
- Params declared in the `params` scope are *config params*
|
||||
- If a config setting references an undeclared param, report an error
|
||||
- Params assigned in a profile are also marked as config params -- they should be used to overwrite existing params or potential script params
|
||||
- CLI params override config params
|
||||
|
||||
3. Execute script, resolve `params` block
|
||||
- CLI params and config params override default values in `params` block
|
||||
- If a required script param is undefined, report an error
|
||||
- If a CLI param is not declared in the `params` block and is not a config param, report an error
|
||||
|
||||
In other words, params are applied in the following order (lowest to highest precedence):
|
||||
|
||||
1. Default value in the `params` block
|
||||
2. Config file (`params { param = value }`)
|
||||
3. Params file (`-params-file params.json`)
|
||||
4. Command-line arguments (`--param value`)
|
||||
|
||||
Any parameter supplied via command line or params file must be declared in the script or config. Supplying an undeclared parameter is an error.
|
||||
|
||||
## Links
|
||||
|
||||
- Community issue: [#4669](https://github.com/nextflow-io/nextflow/issues/4669)
|
||||
- [Workflow outputs ADR](./20251020-workflow-outputs.md)
|
||||
- [Record types ADR](./20260306-record-types.md)
|
||||
|
||||
## Appendix
|
||||
|
||||
### Runtime type analysis via reflection
|
||||
|
||||
Validating and converting params against declared types requires the type annotations to be fully available at runtime. Parameterized types such as `List<String>` must provide both the type (`List`) and the generic type arguments (`[String]`).
|
||||
|
||||
During compilation, type annotations are modeled using `ClassNode`, which provides the "raw" type and type arguments via `getTypeClass() -> Class` and `getGenericsTypes() -> GenericsType[]`.
|
||||
|
||||
At runtime, type annotations are modeled using `Type`, for which there are two primary cases:
|
||||
|
||||
- If the type is parameterized, it is a `ParameterizedType`, which provides the "raw" type and type arguments via `getRawType() -> Class` and `getActualTypeArguments() -> Type[]`.
|
||||
|
||||
- Otherwise, the type is a `Class` corresponding to the raw type.
|
||||
|
||||
This type information can be obtained at runtime from the following entities:
|
||||
|
||||
- Class fields via `Field::getGenericType() -> Type`
|
||||
- Method parameters via `Parameter::getParameterizedType() -> Type`
|
||||
|
||||
For this reason, the `params` block is compiled as a class, so that each parameter declaration is a field which can model a parameterized type.
|
||||
|
||||
Type annotations can be marked as nullable using the `?` suffix. This marker is compiled as a custom `@Nullable` annotation on the corresponding field, so that the runtime can use this information.
|
||||
|
||||
For example, when loading a JSON file as a collection of records, Nextflow uses the given record type to validate each JSON object in the collection:
|
||||
|
||||
- String values that map to a record field with type `Path` are converted to Path values
|
||||
- If a JSON object is missing a record field that is marked as nullable, it is considered valid
|
||||
|
||||
While type annotations are used only at compile-time in all other contexts, they are needed at runtime for pipeline parameters in order to validate and convert external input data to the expected type.
|
||||
154
nextflow/adr/20250922-plugin-spec.md
Normal file
154
nextflow/adr/20250922-plugin-spec.md
Normal file
@@ -0,0 +1,154 @@
|
||||
# Plugin Spec
|
||||
|
||||
- Authors: Ben Sherman
|
||||
- Status: accepted
|
||||
- Deciders: Ben Sherman, Paolo Di Tommaso
|
||||
- Date: 2025-09-22
|
||||
- Tags: plugins
|
||||
|
||||
## Summary
|
||||
|
||||
Provide a way for external systems to understand key information about third-party plugins.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Nextflow plugins need a way to statically declare extensions to the Nextflow language so that external systems can extract information about a plugin without loading it in the JVM.
|
||||
|
||||
Primary use cases:
|
||||
|
||||
- The Nextflow language server needs to know about any config scopes, custom functions, etc, defined by a plugin, in order to recognize them in Nextflow scripts and config files.
|
||||
|
||||
- The Nextflow plugin registry (or other user interfaces) can use this information to provide API documentation.
|
||||
|
||||
## Goals or Decision Drivers
|
||||
|
||||
- External systems (e.g. language server) need to be able to understand plugins without having to load them in the JVM.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Defining specs for the core runtime and core plugins: these definitions are handled separately, although they may share some functionality with plugin specs.
|
||||
|
||||
## Considered Options
|
||||
|
||||
### Nextflow plugin system
|
||||
|
||||
Require external systems to use Nextflow's plugin system to load plugins at runtime in order to extract information about them.
|
||||
|
||||
- **Pro:** Allows any information to be extracted since the entire plugin is loaded
|
||||
|
||||
- **Con:** Requires the entire Nextflow plugin system to be reused or reimplemented. Not ideal for Java applications since the plugin system is implemented in Groovy, incompatible with non-JVM applications
|
||||
|
||||
- **Con:** Requires plugins to be downloaded, cached, loaded in the JVM, even though there is no need to use the plugin.
|
||||
|
||||
### Plugin spec
|
||||
|
||||
Define a plugin spec for every plugin release which is stored and served by the plugin registry as JSON.
|
||||
|
||||
- **Pro:** Allows any system to inspect plugin definitions through a standard JSON document, instead of downloading plugins and loading them into a JVM.
|
||||
|
||||
- **Con:** Requires the plugin spec to be generated at build-time and stored in the plugin registry.
|
||||
|
||||
- **Con:** Requires a standard format to ensure interoperability across different versions of Nextflow, the language server, and third-party plugins.
|
||||
|
||||
## Solution
|
||||
|
||||
Define a plugin spec for every plugin release which is stored and served by the plugin registry as JSON.
|
||||
|
||||
- Plugin developers only need to define [extension points](https://nextflow.io/docs/latest/plugins/developing-plugins.html#extension-points) as usual, and the Gradle plugin will extract the plugin spec and store it in the plugin registry as part of each plugin release.
|
||||
|
||||
- The language server can infer which third-party plugins are required from the `plugins` block in a config file. It will retrieve the appropriate plugin specs from the plugin registry.
|
||||
|
||||
A plugin spec consists of a list of *definitions*. Each definition has a *type* and a *spec*.
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/v1/schema.json",
|
||||
"definitions": [
|
||||
{
|
||||
"type": "ConfigScope",
|
||||
"spec": {
|
||||
// ...
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "Function",
|
||||
"spec": {
|
||||
// ...
|
||||
}
|
||||
},
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The following types of definitions are allowed:
|
||||
|
||||
**ConfigScope**
|
||||
|
||||
Defines a top-level config scope. The spec consists of a *name*, an optional *description*, and *children*.
|
||||
|
||||
The children should be a list of definitions corresponding to nested config scopes and options. The following definitions are allowed:
|
||||
|
||||
- **ConfigOption**: Defines a config option. The spec consists of a *description* and *type*.
|
||||
|
||||
- **ConfigScope**: Defines a nested config scope, using the same spec as for top-level scopes.
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "ConfigScope",
|
||||
"spec": {
|
||||
"name": "hello",
|
||||
"description": "The `hello` scope controls the behavior of the `nf-hello` plugin.",
|
||||
"children": [
|
||||
{
|
||||
"type": "ConfigOption",
|
||||
"spec": {
|
||||
"name": "message",
|
||||
"description": "Message to print to standard output when the plugin is enabled.",
|
||||
"type": "String"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Factory**
|
||||
|
||||
Defines a channel factory that can be included in Nextflow scripts. The spec is the same as for functions.
|
||||
|
||||
**Function**
|
||||
|
||||
Defines a function that can be included in Nextflow scripts. The spec consists of a *name*, an optional *description*, a *return type*, and a list of *parameters*. Each parameter consists of a *name* and a *type*.
|
||||
|
||||
Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "Function",
|
||||
"spec": {
|
||||
"name": "sayHello",
|
||||
"description": "Say hello to the given target",
|
||||
"returnType": "void",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "target",
|
||||
"type": "String"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Operator**
|
||||
|
||||
Defines a channel operator that can be included in Nextflow scripts. The spec is the same as for functions.
|
||||
|
||||
## Rationale & discussion
|
||||
|
||||
Now that there is a Gradle plugin for building Nextflow plugins and a registry to publish and retrieve plugins, it is possible to generate, publish, and retrieve plugin specs in a way that is transparent to plugin developers.
|
||||
|
||||
Plugins specs adhere to a pre-defined [schema](https://raw.githubusercontent.com/nextflow-io/schemas/main/plugin/v1/schema.json) to ensure consistency across different versions of Nextflow. In the future, new versions of the schema can be defined as needed to support new behaviors or requirements.
|
||||
84
nextflow/adr/20250929-repository-directory-traversal.md
Normal file
84
nextflow/adr/20250929-repository-directory-traversal.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# ADR: Repository Directory Traversal API
|
||||
|
||||
**Date**: 2025-09-29
|
||||
**Status**: Accepted
|
||||
**Context**: Need for standardized directory listing across Git hosting providers
|
||||
|
||||
## Decision
|
||||
|
||||
Introduce a `listDirectory(String path, int depth)` method to the `RepositoryProvider` abstraction to enable unified directory traversal across different Git hosting platforms.
|
||||
|
||||
## Context
|
||||
|
||||
Nextflow requires the ability to explore repository directory structures across multiple Git hosting providers (GitHub, GitLab, Bitbucket, Azure DevOps, Gitea) without full repository clones. Each provider has different API capabilities and constraints for directory listing operations.
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
### Core Algorithm
|
||||
|
||||
All providers follow a consistent pattern:
|
||||
1. **Path Resolution**: Normalize path to provider API format
|
||||
2. **Strategy Selection**: Choose recursive vs iterative approach based on API capabilities
|
||||
3. **HTTP Request**: Execute provider-specific API calls
|
||||
4. **Response Processing**: Parse to standardized `RepositoryEntry` objects
|
||||
5. **Depth Filtering**: Apply client-side limits when APIs lack precise depth control
|
||||
|
||||
### API Strategy Classification
|
||||
|
||||
**Strategy A: Native Recursive (GitHub, GitLab, Azure)**
|
||||
- Single HTTP request with recursive parameters
|
||||
- Server-side tree traversal
|
||||
- Performance: O(1) API calls
|
||||
|
||||
**Strategy B: Iterative Traversal (Bitbucket Server, Gitea)**
|
||||
- Multiple HTTP requests per directory level
|
||||
- Client-side recursion management
|
||||
- Performance: O(n) API calls where n = number of directories
|
||||
|
||||
**Strategy C: Limited Support (Bitbucket Cloud)**
|
||||
- Single-level listing only
|
||||
- Throws exceptions for depth > 1
|
||||
|
||||
### Provider Implementation Details
|
||||
|
||||
| Provider | Endpoint | Recursive Support | Performance |
|
||||
|----------|----------|-------------------|-------------|
|
||||
| GitHub | `/git/trees/{sha}?recursive=1` | Native | Optimal |
|
||||
| GitLab | `/repository/tree?recursive=true` | Native | Optimal |
|
||||
| Azure | `/items?recursionLevel=Full` | Native | Optimal |
|
||||
| Bitbucket Server | `/browse/{path}` | Manual iteration | Multiple calls |
|
||||
| Gitea | `/contents/{path}` | Manual iteration | Multiple calls |
|
||||
| Bitbucket Cloud | `/src/{commit}/{path}` | None | Unsupported |
|
||||
|
||||
### HTTP API Constraints
|
||||
|
||||
- **Rate Limiting**: 60-5000 requests/hour depending on provider and authentication
|
||||
- **Response Size**: Controlled by `NXF_GIT_RESPONSE_MAX_LENGTH` environment variable
|
||||
- **Timeouts**: 60-second connect timeout across all providers
|
||||
- **Authentication**: Required for private repositories and higher rate limits
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- **Unified Interface**: Consistent API across all Git hosting providers
|
||||
- **Performance Optimization**: Uses native recursive APIs where available
|
||||
- **Graceful Degradation**: Falls back to iterative traversal when needed
|
||||
- **Error Resilience**: Handles partial failures and API limitations
|
||||
|
||||
### Negative
|
||||
- **Provider Inconsistency**: Performance varies significantly between providers
|
||||
- **API Rate Limits**: Multiple calls required for some providers may hit limits faster
|
||||
- **Memory Usage**: Large directory structures loaded entirely into memory
|
||||
|
||||
### Neutral
|
||||
- **Complexity**: Abstraction layer adds code complexity but improves maintainability
|
||||
- **Testing**: Comprehensive test coverage required for each provider implementation
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Local Git repositories use JGit TreeWalk for optimal performance
|
||||
- Client-side depth filtering ensures consistent behavior across providers
|
||||
- Error handling varies by provider: some return empty lists, others throw exceptions
|
||||
- Future enhancements could include caching based on commit SHA and pagination support
|
||||
|
||||
This decision enables Nextflow to efficiently explore repository structures regardless of the underlying Git hosting platform, with automatic optimization based on each provider's API capabilities.
|
||||
404
nextflow/adr/20251017-typed-processes.md
Normal file
404
nextflow/adr/20251017-typed-processes.md
Normal file
@@ -0,0 +1,404 @@
|
||||
# Typed processes
|
||||
|
||||
- Authors: Ben Sherman
|
||||
- Status: accepted
|
||||
- Deciders: Ben Sherman, Paolo Di Tommaso
|
||||
- Date: 2025-10-17
|
||||
- Tags: lang, static-types, processes
|
||||
|
||||
## Updates
|
||||
|
||||
### Version 1.1 (2026-03-23)
|
||||
|
||||
- Changed the method signature for `stageAs` from `(filePattern, value)` to `(value, filePattern)` to mirror commands like `cp`, `mv`, etc.
|
||||
|
||||
- Replaced annotation-based tuple syntax (`(...): Tuple<...>`) with destructuring syntax (`record(...)`) for better continuity with legacy syntax and record input syntax.
|
||||
|
||||
## Summary
|
||||
|
||||
Support static typing for process inputs and outputs.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The legacy process syntax uses *qualifiers* to describe both the type and staging behavior of each input and output:
|
||||
|
||||
```groovy
|
||||
process FASTQC {
|
||||
input:
|
||||
tuple val(id), path(fastq_1), path(fastq_2)
|
||||
path index
|
||||
|
||||
output:
|
||||
tuple val(id), path("fastqc_${id}_logs")
|
||||
|
||||
script:
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
This syntax has several limimtations:
|
||||
|
||||
- **No static typing**: The `val` qualifier can not specify a type, so there is no way to validate input values. The `path` qualifier can not distinguish between a file and a file collection. The `arity` option was introduced to address this ambiguity, but it is cumbersome and rarely used.
|
||||
|
||||
- **Type and staging behavior are coupled**: Qualifiers like `path` describe both the type *and* the staging behavior (link into task directory). There is no way to specify staging behavior separately, such as staging a tuple element or record field as an environment variable.
|
||||
|
||||
- **No nullability**: There is no way to declare that an input may be `null`. The `path` qualifier raises a runtime error if a null value is received. Outputs can be marked optional, but optional outputs are handled by emitting nothing rather than emitting `null`. A tuple output can be optional, but a tuple element can not.
|
||||
|
||||
- **Limited output expressiveness**: Outputs must be expressed in terms of qualifiers that mirror the input qualifiers. It is difficult to express many kinds of output values, and it is unclear to the user whether a given expression is valid or not.
|
||||
|
||||
## Goals
|
||||
|
||||
- Provide a way to model process inputs and outputs with types from the Nextflow standard library.
|
||||
|
||||
- Separate the *type* of an input from its *staging behavior*.
|
||||
|
||||
- Provide first-class support for nullable inputs and outputs.
|
||||
|
||||
- Allow outputs to be arbitrary expressions, ensuring consistency with the rest of the language.
|
||||
|
||||
- Enable compile-time type checking for processes.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Removing the legacy qualifier syntax -- legacy processes must continue to work without modification.
|
||||
|
||||
- Enforcing type checking -- static type checking will be introduced progressively as an opt-in feature.
|
||||
|
||||
## Decision
|
||||
|
||||
Introduce **typed processes**, which use a new syntax for inputs and outputs based on type annotations instead of qualifiers.
|
||||
|
||||
All other process sections (directives, script, stub, etc) are supported by typed processes without changes. Only the `input:` and `output:` sections are changed.
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
### Typed inputs
|
||||
|
||||
Each input is declared as `name: Type`:
|
||||
|
||||
```groovy
|
||||
process FASTQC {
|
||||
input:
|
||||
id: String
|
||||
fastq_1: Path
|
||||
fastq_2: Path
|
||||
|
||||
script:
|
||||
"""
|
||||
fastqc -o fastqc_${id}_logs ${fastq_1} ${fastq_2}
|
||||
"""
|
||||
}
|
||||
```
|
||||
|
||||
All standard library types except `Channel` and `Value` are valid input types. Inputs of type `Path` (or `Path` collections such as `Set<Path>`) are automatically staged into the task directory.
|
||||
|
||||
### Nullable inputs
|
||||
|
||||
Appending `?` to a type annotation allows the input to be `null`:
|
||||
|
||||
```groovy
|
||||
process CAT_OPT {
|
||||
input:
|
||||
input: Path?
|
||||
|
||||
stage:
|
||||
stageAs input, 'input.txt'
|
||||
|
||||
output:
|
||||
stdout()
|
||||
|
||||
script:
|
||||
'''
|
||||
[[ -f input.txt ]] && cat input.txt || echo 'empty input'
|
||||
'''
|
||||
}
|
||||
```
|
||||
|
||||
By default, a task fails if any input receives `null`.
|
||||
|
||||
### Stage directives
|
||||
|
||||
Staging behavior is moved to a dedicated `stage:` section that appears after `input:`. This replaces the staging aspects of legacy qualifiers:
|
||||
|
||||
| Legacy qualifier | Stage directive |
|
||||
|-------------------|--------------------|
|
||||
| `env('NAME')` | `env 'NAME', value` |
|
||||
| `stdin` | `stdin value` |
|
||||
| `path('name.fa')` | `stageAs file, 'name.fa'` |
|
||||
|
||||
For example:
|
||||
|
||||
```groovy
|
||||
process BLAST {
|
||||
input:
|
||||
fasta: Path
|
||||
|
||||
stage:
|
||||
stageAs fasta, 'query.fa'
|
||||
|
||||
script:
|
||||
"""
|
||||
blastp -query query.fa -db nr
|
||||
"""
|
||||
}
|
||||
```
|
||||
|
||||
Separating staging from type declaration keeps the inputs clean and makes it easier to specify staging behavior independently of the input type.
|
||||
|
||||
### Tuple inputs
|
||||
|
||||
Tuples are declared inline using `tuple(name: Type, ...)`:
|
||||
|
||||
```groovy
|
||||
process FASTQC {
|
||||
input:
|
||||
tuple(id: String, fastq_1: Path, fastq_2: Path)
|
||||
|
||||
script:
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
Each component is destructured into a local variable. This mirrors the `tuple()` constructor used in the output section and in workflow logic, making the syntax consistent.
|
||||
|
||||
### Typed outputs
|
||||
|
||||
Each output declaration consists of an optional name and type, and a value expression:
|
||||
|
||||
```groovy
|
||||
process ECHO {
|
||||
input:
|
||||
message: String
|
||||
|
||||
output:
|
||||
out_file: Path = file('message.txt')
|
||||
out_std: String = stdout()
|
||||
|
||||
script:
|
||||
"""
|
||||
echo '${message}' | tee message.txt
|
||||
"""
|
||||
}
|
||||
```
|
||||
|
||||
When there is only one output, the name and type can be omitted:
|
||||
|
||||
```groovy
|
||||
process ECHO {
|
||||
input:
|
||||
message: String
|
||||
|
||||
output:
|
||||
file('message.txt')
|
||||
|
||||
script:
|
||||
"""
|
||||
echo '${message}' > message.txt
|
||||
"""
|
||||
}
|
||||
```
|
||||
|
||||
Outputs can be arbitrary expressions, rather that being restricted to specific qualifiers such as `tuple` and `val`. Special functions such as `file()`, `files()`, `env()`, and `stdout()` can be composed into the desired output structure.
|
||||
|
||||
### Nullable outputs
|
||||
|
||||
By default, the `file()` and `files()` function raise an error if the given file is missing. These functions can be called with `optionel: true` to allow missing files. This way, it is possible to declare a tuple output that contains nullable values:
|
||||
|
||||
```groovy
|
||||
process MAYBE {
|
||||
input:
|
||||
id: String
|
||||
|
||||
output:
|
||||
tuple(id, file('result.txt'))
|
||||
|
||||
script:
|
||||
"""
|
||||
[[ '$id' == 42 ]] && touch result.txt
|
||||
"""
|
||||
}
|
||||
```
|
||||
|
||||
### Topic emissions
|
||||
|
||||
A `topic:` section emits values to topic channels using the `>>` operator:
|
||||
|
||||
```groovy
|
||||
process CAT {
|
||||
input:
|
||||
message: Path
|
||||
|
||||
output:
|
||||
stdout()
|
||||
|
||||
topic:
|
||||
tuple('cat', eval('cat --version')) >> 'versions'
|
||||
|
||||
script:
|
||||
"""
|
||||
cat ${message}
|
||||
"""
|
||||
}
|
||||
```
|
||||
|
||||
Moving topic emissions to a dedicated section allows them to be defined without having to include them in the process outputs.
|
||||
|
||||
## Distinguishing between typed and legacy processes
|
||||
|
||||
Typed processes are gated behind the `nextflow.enable.types` feature flag, in order to distinguish between typed and legacy processes in the language.
|
||||
|
||||
When a script enables this feature flag, its processes are treated as typed processes; otherwise, its processes are treated as legacy processes. This way, typed and legacy processes cannot be mixed in the same script, but they can be used together as long as they are declared in different scripts.
|
||||
|
||||
While typed and legacy processes are syntactically distinct and could theoretically be allowed in the same script, the feature flag helps distinguish typed vs legacy to the reader (whether human or agent).
|
||||
|
||||
## Alternatives
|
||||
|
||||
### Implicit tuple input
|
||||
|
||||
The syntax for typed process inputs aims to be consistent with typed inputs throughout the rest of the language, such as the `params` block and workflow inputs, which use the pattern of `<name>: <type>`. The `tuple` input qualifier does not fit neatly into this pattern, since it specifies multiple tuple *components*:
|
||||
|
||||
```groovy
|
||||
process QUANT {
|
||||
input:
|
||||
tuple(id: String, fastq_1: Path, fastq_2: Path)
|
||||
index: Path
|
||||
|
||||
// ...
|
||||
}
|
||||
|
||||
workflow {
|
||||
ch_samples = channel.of( tuple('1', file('1_1.fq'), file('1_2.fq')) )
|
||||
index = file('index.fa')
|
||||
QUANT(ch_samples, index)
|
||||
}
|
||||
```
|
||||
|
||||
One alternative is to remove tuple inputs altogether and treat the entire `input:` section as an implicit tuple input:
|
||||
|
||||
```groovy
|
||||
process QUANT {
|
||||
input:
|
||||
id: String
|
||||
fastq_1: Path
|
||||
fastq_2: Path
|
||||
index: Path
|
||||
|
||||
// ...
|
||||
}
|
||||
|
||||
workflow {
|
||||
ch_samples = channel.of( tuple('1', file('1_1.fq'), file('1_2.fq')) )
|
||||
index = file('index.fa')
|
||||
QUANT( ch_samples.combine(index) )
|
||||
}
|
||||
```
|
||||
|
||||
With this approach, a process would always be called with a single input, and multiple sources (e.g. `ch_samples` and `index`) would need to be combined into a single input. This could be done explicitly with the `combine` operator or implicitly by the runtime.
|
||||
|
||||
However, this approach would be a significant change to process call semantics, even if only applied to typed processes. It would likely be difficult to validate for processes with many inputs.
|
||||
|
||||
The tuple destructuring syntax makes it possible to migrate legacy processes to typed processes without changing workflow logic or call semantics. While the `tuple(...)` syntax is a deviation from the typed input syntax used by the rest of the language, such deviations can be appropriate and even advantageous when used judiciously in a custom language.
|
||||
|
||||
### Type annotation syntax for tuple inputs
|
||||
|
||||
Another alternative for tuple inputs is to use a type annotation:
|
||||
|
||||
```groovy
|
||||
process FASTQC {
|
||||
input:
|
||||
(id, fastq_1, fastq_2): Tuple<String,Path,Path>
|
||||
index: Path
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
This approach attempts to bring the syntax closer to the `<name>: <type>` pattern while maintaining support for tuple destructuring. This syntax was used in the first preview of typed processes in Nextflow 25.10.
|
||||
|
||||
However, this syntax needlessly separates the component name from its corresponding type, making it harder to read and validate. Although it is semantically equivalent to the legacy syntax, it looks and feels very different, which can be jarring for users.
|
||||
|
||||
With the introduction of records, the `tuple(...)` destructuring syntax emerged as a clear pattern to follow for both records and tuples:
|
||||
|
||||
**Legacy process:**
|
||||
```groovy
|
||||
process FASTQC {
|
||||
input:
|
||||
tuple val(id), path(fastq_1), path(fastq_2)
|
||||
|
||||
output:
|
||||
tuple val(id), path("fastqc_${id}_logs")
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
**Typed process (tuple):**
|
||||
```groovy
|
||||
process FASTQC {
|
||||
input:
|
||||
tuple(id: String, fastq_1: Path, fastq_2: Path)
|
||||
|
||||
output:
|
||||
tuple(id, file("fastqc_${id}_logs"))
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
**Typed process (record):**
|
||||
```groovy
|
||||
process FASTQC {
|
||||
input:
|
||||
record(
|
||||
id: String,
|
||||
fastq_1: Path,
|
||||
fastq_2: Path
|
||||
)
|
||||
|
||||
output:
|
||||
record(
|
||||
id: id,
|
||||
fastqc: file("fastqc_${id}_logs")
|
||||
)
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
This pattern provides the best balance of continuity with the old way and consistency with static typing:
|
||||
|
||||
- A legacy process can be migrated to a typed process by replacing the `tuple` input/output qualifier with the `tuple` destructor/constructor.
|
||||
- A typed process can be migrated from tuples to records by replacing `tuple` with `record` and adding fields to the record output.
|
||||
- The `tuple` and `record` destructors use the same `<name>: <type>` pattern used by the rest of the language.
|
||||
- At each stage, the inputs and outputs mirror each other without creating syntactic confusion.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive:**
|
||||
|
||||
- Type annotations make processes self-documenting and provide the information needed to perform static type checking.
|
||||
|
||||
- Separating type from staging behavior (the `stage:` section) makes each concern independently clear.
|
||||
|
||||
- Nullable types (`?`) provide first-class support for nullable input files.
|
||||
|
||||
- Outputs can be structured arbitrarily and can contain nullable files.
|
||||
|
||||
**Negative:**
|
||||
|
||||
- The `each` qualifier is not supported; pipelines using it must be refactored to use the `combine` operator before migrating to typed processes.
|
||||
|
||||
- The typed syntax must be maintained alongside the legacy syntax, which makes the codebase more complex and may cause confusion as the community transitions to the new syntax.
|
||||
|
||||
**Neutral:**
|
||||
|
||||
- Typed processes use the same standard types as the rest of the language, so no additional type vocabulary is introduced.
|
||||
|
||||
- Typed processes are enabled by a feature flag, which introduces new functionality without breaking existing code and helps distinguish between typed and legacy code.
|
||||
|
||||
## Links
|
||||
|
||||
- [Nextflow standard types](https://nextflow.io/docs/latest/reference/stdlib-types.html)
|
||||
- Community issues: #1694, #2678
|
||||
- Related nf-core discussion: https://github.com/nf-core/modules/issues/4311
|
||||
- Original implementation: #4553
|
||||
198
nextflow/adr/20251020-workflow-outputs.md
Normal file
198
nextflow/adr/20251020-workflow-outputs.md
Normal file
@@ -0,0 +1,198 @@
|
||||
# Workflow outputs
|
||||
|
||||
- Authors: Ben Sherman
|
||||
- Status: accepted
|
||||
- Date: 2025-10-20
|
||||
- Tags: lang, workflows
|
||||
|
||||
## Summary
|
||||
|
||||
Introduce a unified, dataflow-centric way to declare the top-level outputs of a workflow.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
In Nextflow DSL1, each process used `publishDir` to copy output files from the work directory to an external location. Nextflow DSL2 inherited this approach but it became increasingly problematic as pipelines grew larger and more modular:
|
||||
|
||||
- **Mismatch with reusable modules**: Publishing rules often depend on how a process is used in a given pipeline. Setting `publishDir` inside a module process makes the module less reusable, since the publish path and mode are baked into the process definition. Using process selectors in configuration is verbose and fragile.
|
||||
|
||||
- **Fragmented outputs**: Publishing logic is scattered across many module files. There is no single place to see what a pipeline produces or to reason about the output structure.
|
||||
|
||||
- **Redundant configuration**: Common settings like the base output directory and publish mode must be repeated in every `publishDir` declaration, leading to duplication.
|
||||
|
||||
- **Mismatch with channels**: Channels carry both files and structured metadata (e.g., sample IDs, quality flags). The `publishDir` directive matches files with glob patterns and cannot capture metadata unless it happens to be written to a file. This mismatch makes it difficult to produce structured, self-describing outputs.
|
||||
|
||||
## Goals
|
||||
|
||||
- Declare all pipeline outputs in a single location alongside the entry workflow.
|
||||
|
||||
- Assign outputs from channels rather than from individual process definitions, decoupling pipeline-specific publishing rules from reusable modules.
|
||||
|
||||
- Support dynamic and fine-grained file publishing to match common publishing patterns (e.g. directory per sample, directory per pipeline step).
|
||||
|
||||
- Support structured index files (CSV, JSON, YAML) that preserve output files with associated metadata.
|
||||
|
||||
- Define publishing behavior (mode, overwrite, storage class, etc.) globally in the config.
|
||||
|
||||
- Support type annotations on output declarations for documentation and compile-time validation.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Removing support for `publishDir` immediately -- `publishDir` should continue to work without modification, although it may eventually be phased out as users migrate away from it.
|
||||
|
||||
- Publishing outputs from processes or named workflows -- only the entry workflow has a `publish:` section.
|
||||
|
||||
- Defining a JSON schema for workflow outputs -- schema/spec generation will be explored in the future.
|
||||
|
||||
## Decision
|
||||
|
||||
Introduce the `output` block for declaring workflow outputs. Each output defines how files are published to the output directory, and the format of the index file (if defined).
|
||||
|
||||
Introduce the output directory as a first-class concept in Nextflow, as well as the `workflow.output` config scope for controlling publishing behavior.
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
### Output definition
|
||||
|
||||
Workflow outputs consist of an `output` block, which declares each output, and a `publish:` section in the entry workflow, which assigns a dataflow source (channel or value) to each output:
|
||||
|
||||
```groovy
|
||||
workflow {
|
||||
main:
|
||||
ch_fastqc = FASTQC(ch_reads)
|
||||
ch_report = MULTIQC(ch_fastqc.collect())
|
||||
|
||||
publish:
|
||||
fastqc = ch_fastqc
|
||||
report = ch_report
|
||||
}
|
||||
|
||||
output {
|
||||
fastqc: Channel<Path> {
|
||||
path 'fastqc'
|
||||
}
|
||||
report: Path {
|
||||
path '.'
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Every output assigned in `publish:` must be declared in the `output` block, and vice versa. A mismatch is a compile-time error.
|
||||
|
||||
Each output declaration can specify a type annotation for documentation and type checking support. Type annotations are optional and do not change runtime behavior. They are used by the type checker to validate the `publish:` section and the `path` directive.
|
||||
|
||||
### Output directory
|
||||
|
||||
The top-level output directory defaults to `results` in the launch directory. It can be overridden from the command line or config file:
|
||||
|
||||
```bash
|
||||
nextflow run main.nf -output-dir my-results
|
||||
```
|
||||
|
||||
```groovy
|
||||
// nextflow.config
|
||||
outputDir = 'my-results'
|
||||
```
|
||||
|
||||
All publish paths declared in the `output` block are relative to this directory. Absolute paths are not allowed.
|
||||
|
||||
### Static and dynamic publish paths
|
||||
|
||||
The `path` directive accepts a string for a fixed path, or a closure for per-value paths:
|
||||
|
||||
```groovy
|
||||
output {
|
||||
// static: all files go to results/fastq/
|
||||
reads {
|
||||
path 'fastq'
|
||||
}
|
||||
|
||||
// dynamic: results are organized by sample id
|
||||
samples {
|
||||
path { sample -> "${sample.id}" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Nextflow recursively scans channel values for files, including files nested inside lists, maps, records, and tuples. Files that did not originate from the work directory are not published.
|
||||
|
||||
### Fine-grained file publishing with `>>`
|
||||
|
||||
Within a `path` closure, individual files can be published to different locations using the `>>` operator. Only files explicitly captured with `>>` are published; other files in the value are ignored.
|
||||
|
||||
```groovy
|
||||
output {
|
||||
samples {
|
||||
path { sample ->
|
||||
sample.fastqc >> "fastqc/"
|
||||
sample.bam >> (params.save_bams ? "align/" : null)
|
||||
sample.bam_index >> (params.save_bams ? "align/" : null)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The *publish source* (left-hand side) should be a file or collection of files. The *publish target* (right-hand side) should be a relative path. If the target has a trailing slash, then the source is published *into* the target directory; otherwise the source is published *as* the target name.
|
||||
|
||||
A `null` target suppresses publishing for that file, and a `null` source is also a no-op. This way, publishing of individual files can be disabled by either setting the record field to `null` in workflow logic or using a param in the publish statement.
|
||||
|
||||
### Index files
|
||||
|
||||
Each output can generate a structured index file that records each published channel value along with its metadata. Supported formats are CSV, JSON, and YAML.
|
||||
|
||||
```groovy
|
||||
output {
|
||||
samples {
|
||||
path 'fastq'
|
||||
index {
|
||||
path 'samples.csv'
|
||||
header true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The index file is essentially a *samplesheet* -- it preserves the structure of files and metadata in the published channel, and can be easily passed as input to downstream pipelines. Metadata fields (sample IDs, quality flags, etc.) do not need to be written to a separate metadata file or encoded into file paths.
|
||||
|
||||
Files that did not originate from the work directory are not published, but are still included in the index.
|
||||
|
||||
### Global defaults via configuration
|
||||
|
||||
Common publish settings can be set globally under the `workflow.output` config scope:
|
||||
|
||||
```groovy
|
||||
// nextflow.config
|
||||
workflow {
|
||||
output {
|
||||
mode = 'copy'
|
||||
overwrite = 'lenient'
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
These defaults can be overridden per-output in the `output` block:
|
||||
|
||||
```groovy
|
||||
// main.nf
|
||||
output {
|
||||
fastqc {
|
||||
mode = 'symlink'
|
||||
overwrite = true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Alternatives
|
||||
|
||||
### Publishing from processes and subworkflows
|
||||
|
||||
Earlier iterations allowed for workflow outputs to be published from subworkflows or processes, instead of requiring all workflow outputs to be propagated up to the entry workflow.
|
||||
|
||||
While this approach is less verbose, it breaks the modularity of processes and subworkflows. Publishing behavior is inherent to the pipeline, not the individual subcomponents which could be shared across many pipelines. The process or subworkflow should expose all of its outputs as channels, and the calling pipeline should decide whether and how to publish these outputs.
|
||||
|
||||
On the other hand, propagating all workflow outputs to the top will make pipelines more verbose, especially when using "skinny tuple" channels. This issue will be alleviated by migrating from tuples to records -- for this reason, it is recommended that large pipelines be migrated to records before being migrated to workflow outputs.
|
||||
|
||||
## Links
|
||||
|
||||
- Community issues: [#4042](https://github.com/nextflow-io/nextflow/issues/4042), [#4661](https://github.com/nextflow-io/nextflow/issues/4661), [#4670](https://github.com/nextflow-io/nextflow/issues/4670)
|
||||
- [Workflow params ADR](./20250825-workflow-params.md)
|
||||
- [Record types ADR](./20260306-record-types.md)
|
||||
930
nextflow/adr/20251114-module-system.md
Normal file
930
nextflow/adr/20251114-module-system.md
Normal file
@@ -0,0 +1,930 @@
|
||||
# Module System for Nextflow
|
||||
|
||||
- Authors: Paolo Di Tommaso
|
||||
- Status: approved
|
||||
- Date: 2025-01-06
|
||||
- Tags: modules, dsl, registry, versioning, architecture
|
||||
- Version: 2.7
|
||||
|
||||
## Updates
|
||||
|
||||
### Version 2.7 (2026-03-09)
|
||||
- **Renamed `.checksum` to `.module-info`**: Leaves room for additional properties in the future
|
||||
- **Removed `@` prefix from module scopes**: Local modules are distinguished from remote modules by presence/absence of `./` prefix
|
||||
- **Removed version pinning from config**: Installed module versions are now inferred from the `meta.yml` of each module in the `modules/` directory instead of being declared in `nextflow.config`
|
||||
|
||||
### Version 2.6 (2026-01-28)
|
||||
- **Removed module parameters**: Module parameters specification moved to separate spec document.
|
||||
|
||||
### Version 2.5 (2026-01-23)
|
||||
- **Module parameters**: Replaced structured tool arguments with general module parameters defined in `meta.yml`
|
||||
- **Simplified tools section**: Removed `args` property from tools; tool arguments now configured via module parameters
|
||||
- **Simplified `requires` block**: Removed `plugins`, `modules`, and `subworkflows` sub-properties; `requires` now only contains `nextflow` version constraint
|
||||
- **Process modules focus**: Removed sub-workflow references; spec is now focused on process modules only
|
||||
|
||||
### Version 2.4 (2026-01-15)
|
||||
- **Removed transitive dependency resolution**: Module dependencies are explicit only; no automatic transitive resolution
|
||||
- **Removed `freeze` command**: No longer needed without transitive dependency management
|
||||
- **Simplified model**: Each module explicitly declares its dependencies in `nextflow.config`
|
||||
|
||||
### Version 2.3 (2026-01-15)
|
||||
- **Resolution Rules table**: Added clear table specifying behavior for each combination of local state and declared version
|
||||
- **Local modification protection**: Locally modified modules (checksum mismatch) are NOT overridden unless `-force` flag is used
|
||||
- **Simplified storage model**: Single version per module locally (`modules/@scope/name/` without version in path)
|
||||
- **`.checksum` file**: Registry checksum cached locally for fast integrity verification without network calls
|
||||
|
||||
### Version 2.2 (2025-01-06) — *Superseded by v2.5*
|
||||
- **Structured tool arguments**: Added `args` property to `tools` section for type-safe argument configuration
|
||||
- **New implicit variables**: `tools.<toolname>.args.<argname>` returns formatted flag+value; `tools.<toolname>.args` returns all args concatenated
|
||||
- **Deprecation**: All `ext.*` custom directives (e.g., `ext.args`, `ext.args2`, `ext.args3`, `ext.prefix`, `ext.suffix`) deprecated in favor of structured tool arguments
|
||||
- *Note: Tool arguments replaced by module parameters in v2.5*
|
||||
|
||||
### Version 2.1 (2024-12-11)
|
||||
- **Unified dependencies**: Consolidated `components`, `dependencies`, and `requires` into single `requires` field
|
||||
- **Unified version syntax**: `[scope/]name[@constraint]` format across plugins and modules
|
||||
- **Deprecation**: `components` field deprecated (use top-level `modules` instead)
|
||||
|
||||
## Context and Problem Statement
|
||||
|
||||
Nextflow supports local script inclusion via `include` directive but lacks standardized mechanisms for package management, versioning, and distribution of reusable process definitions. This limits code reuse and reproducibility across the ecosystem.
|
||||
|
||||
Discussion/request goes back to at least 2019, see GitHub issues [#1376](https://github.com/nextflow-io/nextflow/issues/1376), [#1463](https://github.com/nextflow-io/nextflow/issues/1463) and [#4122](https://github.com/nextflow-io/nextflow/issues/4112).
|
||||
|
||||
## Decision
|
||||
|
||||
Implement a module system with four core capabilities:
|
||||
|
||||
1. **Remote module inclusion** via registry
|
||||
2. **Semantic versioning** with dependency resolution
|
||||
3. **Unified Nextflow Registry** (rebrand existing Nextflow registry)
|
||||
4. **First-class CLI support** (install, publish, search, list, remove, run)
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
### 1. Remote Module Inclusion
|
||||
|
||||
**DSL Syntax**:
|
||||
```groovy
|
||||
// Include from registry (scoped module name without `./` prefix)
|
||||
include { BWA_ALIGN } from 'nf-core/bwa-align'
|
||||
|
||||
// Existing file-based includes remain supported
|
||||
include { MY_PROCESS } from './modules/my-process.nf'
|
||||
```
|
||||
|
||||
**Module Naming**: Scoped modules `scope/name` (e.g., `nf-core/salmon`, `myorg/custom`). Local paths supported for backwards compatibility. No nested paths with the module are allowed - each module must have a `main.nf` as the entry point.
|
||||
|
||||
**Version Resolution**: Installed module versions are inferred from the `meta.yml` of each module in the `modules/` directory. If a module is not present locally, the latest available version is downloaded from the registry.
|
||||
|
||||
**Resolution Order**:
|
||||
1. Check local `modules/scope/name/` exists
|
||||
2. Verify integrity against `.module-info` file
|
||||
3. Apply resolution rules (see below)
|
||||
|
||||
**Resolution Rules**:
|
||||
|
||||
| Local State | Action |
|
||||
|-------------|--------|
|
||||
| Missing | Download latest from registry |
|
||||
| Exists, checksum valid | Use local module (version from `meta.yml`) |
|
||||
| Exists, checksum mismatch | **Warn**: locally modified, will NOT replace unless `-force` is used |
|
||||
|
||||
**Key Behaviors**:
|
||||
- **Local modification**: When the local module content was manually changed (checksum mismatch with `.module-info`), Nextflow warns and does NOT override to prevent accidental loss of local changes
|
||||
- **Force flag**: Use `-force` with `nextflow module install` to override locally modified modules
|
||||
|
||||
**Resolution Timing**: Modules resolved at workflow parse time (after plugin resolution at startup).
|
||||
|
||||
**Local Storage**: Downloaded modules stored in `modules/scope/name/` directory in project root (not global cache). Each module must contain a `main.nf` file as the required entry point. It is intended that module source code will be committed to the pipeline git repository.
|
||||
|
||||
### 2. Semantic Versioning and Configuration
|
||||
|
||||
**Version Format**: MAJOR.MINOR.PATCH
|
||||
- **MAJOR**: Breaking changes to process signatures, inputs, or outputs
|
||||
- **MINOR**: New processes, backward-compatible enhancements
|
||||
- **PATCH**: Bug fixes, documentation updates
|
||||
|
||||
**Registry Configuration** (`nextflow.config`):
|
||||
```groovy
|
||||
registry {
|
||||
url = 'https://registry.nextflow.io' // Default registry
|
||||
|
||||
// allow the use of multiple registry url for resolving module
|
||||
// across custom registries, e.g.
|
||||
// url = [ 'https://custom.registry.com', 'https://registry.nextflow.io' ]
|
||||
|
||||
auth {
|
||||
'registry.nextflow.io' = '${NXF_REGISTRY_TOKEN}'
|
||||
'npm.myorg.com' = '${MYORG_TOKEN}'
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Module Spec** (`meta.yml`):
|
||||
```yaml
|
||||
name: nf-core/bwa-align
|
||||
version: 1.2.4 # This module's version
|
||||
|
||||
requires:
|
||||
nextflow: ">=24.04.0"
|
||||
```
|
||||
|
||||
**Version Constraints** (unified `name@constraint` syntax):
|
||||
- `name`: Any version (latest)
|
||||
- `name@1.2.3`: Exact version
|
||||
- `name@>=1.2.3`: Greater or equal
|
||||
- `name@>=1.2.3,<2.0.0`: Range (comma-separated)
|
||||
|
||||
**Version Notation Consistency**:
|
||||
|
||||
Modules use the same version constraint syntax already supported by both `nextflowVersion` and plugins:
|
||||
|
||||
| Notation | Meaning | nextflowVersion | Plugins | Modules |
|
||||
| :---- | :---- | :---- | :---- | :---- |
|
||||
| 1.2.3 | Exact version | ✓ | ✓ | ✓ |
|
||||
| >=1.2.3 | Greater or equal | ✓ | ✓ | ✓ |
|
||||
| <=1.2.3 | Less or equal | ✓ | ✓ | ✓ |
|
||||
| >1.2.3 | Greater than | ✓ | ✓ | ✓ |
|
||||
| <1.2.3 | Less than | ✓ | ✓ | ✓ |
|
||||
| >=1.2, <2.0 | Range (comma) | ✓ | ✓ | ✓ |
|
||||
| !=1.2.3 | Not equal | ✓ | - | - |
|
||||
| 1.2+ | >=1.2.x <2.0 | ✓ | - | - |
|
||||
| 1.2.+ | >=1.2.0 <1.3.0 | ✓ | - | - |
|
||||
| ~1.2.3 | >=1.2.3 <1.3.0 | - | ✓ | - |
|
||||
|
||||
Using comparison operators (`>=`, `<`) with comma-separated ranges provides the same expressive power as
|
||||
npm-style `^` and `~` notation while maintaining consistency with existing Nextflow version constraint syntax.
|
||||
This avoids introducing new notation that would require additional parser support.
|
||||
|
||||
**Module Resolution**:
|
||||
|
||||
Installed module versions are inferred from the `meta.yml` file for each module in the `modules/` directory.
|
||||
|
||||
### 3. Unified Nextflow Registry
|
||||
|
||||
**Architecture Decision**: Extend existing Nextflow registry at `registry.nextflow.io` to host both plugins and modules.
|
||||
|
||||
**Current Plugin API** (reference: https://registry.nextflow.io/openapi/):
|
||||
```
|
||||
GET /api/v1/plugins # List/search plugins
|
||||
GET /api/v1/plugins/{pluginId} # Get plugin + all releases
|
||||
GET /api/v1/plugins/{pluginId}/{version} # Get specific release
|
||||
GET /api/v1/plugins/{pluginId}/{version}/download/{fileName} # Download artifact
|
||||
POST /api/v1/plugins/release # Create draft release
|
||||
POST /api/v1/plugins/release/{releaseId}/upload # Upload artifact
|
||||
```
|
||||
|
||||
**Module API** (reference: https://github.com/seqeralabs/plugin-registry/pull/266):
|
||||
```
|
||||
GET /api/modules?query=<text> # Search modules (semantic search)
|
||||
GET /api/modules/{name} # Get module + latest release
|
||||
GET /api/modules/{name}/releases # List all releases
|
||||
GET /api/modules/{name}/{version} # Get specific release
|
||||
GET /api/modules/{name}/{version}/download # Download module bundle
|
||||
POST /api/modules/{name} # Publish module version (authenticated)
|
||||
```
|
||||
|
||||
Note: The `{name}` parameter includes the namespace prefix (e.g., "nf-core/fastqc").
|
||||
|
||||
**Registry URL**: `registry.nextflow.io`
|
||||
|
||||
**Artifact Types**:
|
||||
- **Plugins**: JAR files with JSON metadata, resolved at startup
|
||||
- **Modules**: Source archives (.nf + meta.yml), resolved at parse time
|
||||
|
||||
**Benefits**:
|
||||
- Reuses existing infrastructure (HTTP service, S3 storage, authentication)
|
||||
- Consistent API patterns for both artifact types
|
||||
- Operational simplicity (one service vs. two)
|
||||
- Internal module API already partially implemented
|
||||
|
||||
### 4. First-Class CLI Support
|
||||
|
||||
**Commands**:
|
||||
```bash
|
||||
nextflow module run scope/name # Run a module directly without a wrapper script
|
||||
nextflow module search <query> # Search registry
|
||||
nextflow module install scope/name # Install a module
|
||||
nextflow module list # Show installed vs configured
|
||||
nextflow module remove scope/name # Remove from config + local cache
|
||||
nextflow module publish scope/name # Publish to registry (requires api key)
|
||||
```
|
||||
|
||||
**General Notes**:
|
||||
- All commands respect the `registry.url` configuration for custom registries
|
||||
|
||||
#### `nextflow module run scope/name`
|
||||
|
||||
Run a module directly without requiring a wrapper workflow script. This command enables standalone execution of any module by automatically mapping command-line arguments to the module's process inputs. If the module is not available locally, it is automatically installed before execution.
|
||||
|
||||
**Arguments**:
|
||||
- `scope/name`: Module identifier to run (required)
|
||||
|
||||
**Options**:
|
||||
- `-version <ver>`: Run a specific version (default: latest or configured version)
|
||||
- `--<input_name> <value>`: Map value to the corresponding module process input channel
|
||||
- All standard `nextflow run` options (e.g., `-profile`, `-work-dir`, `-resume`, etc.)
|
||||
|
||||
**Behavior**:
|
||||
1. Checks if module is installed locally; if not, downloads from registry
|
||||
2. Parses the module's `main.nf` to identify the main process and its input declarations
|
||||
3. Validates command-line arguments against the process input declarations
|
||||
4. Generates an implicit workflow that wires CLI arguments to process inputs
|
||||
5. Executes the workflow using standard Nextflow runtime
|
||||
|
||||
**Input Mapping**:
|
||||
- Named arguments (`--reads`, `--reference`) are mapped to corresponding process inputs
|
||||
- File paths are automatically converted to files for process file inputs
|
||||
- Multiple values can be provided for inputs expecting collections
|
||||
- Required inputs without defaults must be provided; optional inputs use declared defaults
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
# Run BWA alignment module with input files
|
||||
nextflow module run nf-core/bwa-align \
|
||||
--reads 'samples/*_{1,2}.fastq.gz' \
|
||||
--reference genome.fa
|
||||
|
||||
# Run a specific version with Nextflow options
|
||||
nextflow module run nf-core/fastqc -version 1.0.0 \
|
||||
--input 'data/*.fastq.gz' \
|
||||
-profile docker \
|
||||
-resume
|
||||
|
||||
# Run with work directory and output specification
|
||||
nextflow module run nf-core/salmon \
|
||||
--reads reads.fq \
|
||||
--index salmon_index \
|
||||
-work-dir /tmp/work \
|
||||
-output-dir results/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `nextflow module search <query>`
|
||||
|
||||
Search the Nextflow registry for available modules matching the specified query. The search operates against module names, descriptions, tags, and author information. Results are displayed with module name, latest version, description, and download statistics.
|
||||
|
||||
**Arguments**:
|
||||
- `<query>`: Search term (required) - matches against module metadata
|
||||
|
||||
**Options**:
|
||||
- `-limit <n>`: Maximum number of results to return (default: 10)
|
||||
- `-json`: Output results in JSON format for programmatic use
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
nextflow module search bwa
|
||||
nextflow module search "alignment" -limit 50
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `nextflow module install <scope/name>`
|
||||
|
||||
Download and install a module to the local `modules/` directory.
|
||||
|
||||
**Arguments**:
|
||||
- `<scope/name>`: Module identifier.
|
||||
|
||||
**Options**:
|
||||
- `-version <ver>`: Install a specific version (default: latest)
|
||||
- `-force`: Overwrite any local changes
|
||||
|
||||
**Behavior**:
|
||||
1. If `-version` not specified, queries registry for the latest available version
|
||||
2. Checks if local module exists and verifies integrity against `.module-info` file
|
||||
3. If local module is unmodified and version differs: replaces with requested version
|
||||
4. If local module was modified (checksum mismatch): warns and aborts unless `-force` is used
|
||||
5. Downloads the module archive from the registry
|
||||
6. Extracts to `modules/scope/name/` directory
|
||||
7. Stores `.module-info` file from registry's X-Checksum response header
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
nextflow module install nf-core/bwa-align # Install specific module (latest)
|
||||
nextflow module install nf-core/salmon -version 1.2.0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `nextflow module list`
|
||||
|
||||
Display the status of all modules, comparing what is configured in `nextflow.config` against what is actually installed in the `modules/` directory.
|
||||
|
||||
**Options**:
|
||||
- `-json`: Output in JSON format
|
||||
- `-outdated`: Only show modules with available updates
|
||||
|
||||
**Output columns**:
|
||||
- Module name (`scope/name`)
|
||||
- Installed version (from `modules/scope/name/meta.yml`)
|
||||
- Latest available version (from registry)
|
||||
- Status indicator (up-to-date, outdated, missing)
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
nextflow module list
|
||||
nextflow module list -outdated
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `nextflow module remove scope/name`
|
||||
|
||||
Remove a module from the local `modules/` directory.
|
||||
|
||||
**Arguments**:
|
||||
- `scope/name`: Module identifier to remove (required)
|
||||
|
||||
**Options**:
|
||||
- `-keep-files`: Remove `.module-info` file but keep local files
|
||||
|
||||
**Behavior**:
|
||||
1. Removes the module directory from `modules/scope/name/`
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
nextflow module remove nf-core/bwa-align
|
||||
nextflow module remove myorg/custom -keep-files
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `nextflow module publish scope/name`
|
||||
|
||||
Publish a module to the Nextflow registry, making it available for others to install. Requires authentication via API key and appropriate permissions for the target scope.
|
||||
|
||||
**Arguments**:
|
||||
- `scope/name`: Module identifier to publish (required)
|
||||
|
||||
**Options**:
|
||||
- `-registry <url>`: Target registry URL (default: `registry.nextflow.io`)
|
||||
- `-tag <tag>`: Additional tags for discoverability
|
||||
- `-dry-run`: Validate without publishing
|
||||
|
||||
**Behavior**:
|
||||
1. Validates `meta.yml` schema and required fields (name, version, description)
|
||||
2. Verifies that `main.nf` exists and is valid Nextflow syntax
|
||||
3. Verifies that `README.md` documentation is present
|
||||
4. Authenticates with registry using configured credentials
|
||||
5. Creates a release draft and uploads the module archive
|
||||
6. Publishes the release, making it available for installation
|
||||
|
||||
**Requirements**:
|
||||
- Valid `meta.yml` with name, version, and description
|
||||
- `main.nf` entry point file
|
||||
- `README.md` documentation
|
||||
- Authentication token configured in `registry.auth` or `NXF_REGISTRY_TOKEN`
|
||||
- Write permission for the target scope
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
nextflow module publish myorg/my-process
|
||||
nextflow module publish myorg/my-process -dry-run
|
||||
```
|
||||
|
||||
## Module Structure
|
||||
|
||||
**Directory Layout**:
|
||||
Everything within the module directory should be uploaded. Module bundle should not exceed 1MB (uncompressed). Typically this is expected to look something like this:
|
||||
```
|
||||
my-module/
|
||||
├── main.nf # Required: entry point for module
|
||||
├── meta.yml # Required: Module spec (version, metadata, I/O specs)
|
||||
├── README.md # Required: Module description
|
||||
└── tests/ # Optional tests
|
||||
```
|
||||
|
||||
**Module Spec extension** (`meta.yml`):
|
||||
```yaml
|
||||
name: nf-core/bwa-align
|
||||
version: 1.2.4 # This module's version
|
||||
description: Align reads using BWA-MEM
|
||||
authors:
|
||||
- nf-core community
|
||||
license: MIT
|
||||
|
||||
requires:
|
||||
nextflow: ">=24.04.0"
|
||||
```
|
||||
|
||||
**Local Storage Structure**:
|
||||
```
|
||||
project-root/
|
||||
├── nextflow.config
|
||||
├── main.nf
|
||||
└── modules/ # Local module cache
|
||||
├── nf-core/
|
||||
│ ├── bwa-align/
|
||||
│ │ ├── .module-info # Cached registry checksum
|
||||
│ │ ├── meta.yml
|
||||
│ │ └── main.nf # Required entry point
|
||||
│ └── samtools/view/
|
||||
│ ├── .module-info
|
||||
│ ├── meta.yml
|
||||
│ └── main.nf # Required entry point
|
||||
└── myorg/
|
||||
└── custom-process/
|
||||
├── .module-info
|
||||
├── meta.yml
|
||||
└── main.nf # Required entry point
|
||||
```
|
||||
|
||||
**Module Integrity Verification**:
|
||||
- On install: `.module-info` file created from registry's X-Checksum response header
|
||||
- On run: Local module checksum compared against `.module-info` file
|
||||
- If match: Proceed without network call
|
||||
- If mismatch: Report warning (module may have been locally modified)
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
**Phase 1**: Module schema, local module loading, validation tools
|
||||
|
||||
**Phase 2**: Extend Nextflow registry for modules, implement caching, add `install` and `search` commands
|
||||
|
||||
**Phase 3**: Extend DSL parser for `from module` syntax
|
||||
|
||||
**Phase 4**: Implement `publish` command with authentication and `run` command
|
||||
|
||||
**Phase 5**: Advanced features (search UI, language server integration, ontology validation)
|
||||
|
||||
## Technical Details
|
||||
|
||||
**Module Resolution Flow**:
|
||||
1. Parse `include` statements → extract module names (e.g., `nf-core/bwa-align`)
|
||||
2. For each module:
|
||||
a. Check local `modules/scope/name/` exists
|
||||
- If exists → read installed version from `modules/scope/name/meta.yml`
|
||||
- If missing → download latest version from registry
|
||||
b. Verify local module integrity against `.module-info` file
|
||||
- Checksum mismatch → warn and do NOT override (local changes detected)
|
||||
3. On download: store module to `modules/scope/name/` with `.module-info` file
|
||||
4. Read `meta.yml` file: Validates Nextflow requirement → Fail if not fulfilled
|
||||
5. Parse module's `main.nf` file → make processes available
|
||||
|
||||
**Security**:
|
||||
- SHA-256 checksum verification on download (stored in `.module-info` file)
|
||||
- Integrity verification on run (local checksum vs `.module-info` file)
|
||||
- Authentication required for publishing
|
||||
- Support for private registries
|
||||
|
||||
**Integration with Plugin System**:
|
||||
- Both plugins and modules query same registry
|
||||
- Single authentication system
|
||||
- Separate cache locations: `$NXF_HOME/plugins/` (global) vs `modules/` (per-project)
|
||||
|
||||
## Comparison: Plugins vs. Modules
|
||||
|
||||
| Aspect | Plugins | Modules |
|
||||
|--------|---------|---------|
|
||||
| Purpose | Extend runtime | Reusable processes |
|
||||
| Format | JAR files | Source code (.nf) |
|
||||
| Resolution | Startup | Parse time |
|
||||
| Metadata | JSON spec | YAML spec |
|
||||
| Naming | `nf-amazon` | `nf-core/salmon` |
|
||||
| Cache Location | `$NXF_HOME/plugins/` | `modules/scope/name/` |
|
||||
| Version Config | `plugins {}` in config | `meta.yml` in `modules/` directory |
|
||||
| Registry Path | `/api/v1/plugins/` | `/api/modules/{name}` |
|
||||
|
||||
## Rationale
|
||||
|
||||
**Why unified registry?**
|
||||
- Reuses battle-tested infrastructure (HTTP API, S3, auth)
|
||||
- Single discovery experience for ecosystem
|
||||
- Lower operational overhead
|
||||
- Type-specific handling maintains separation of concerns
|
||||
|
||||
**Why infer versions from `meta.yml` instead of pinning in a separate file?**
|
||||
- Simple: install a version once and it is captured in the module files
|
||||
- Reproducibility via committing the `modules/` directory (including `meta.yml`) to the project git repository
|
||||
- Reduces configuration burden: no need to keep config in sync with installed state
|
||||
|
||||
**Why parse-time resolution?**
|
||||
- Modules are source code, not compiled artifacts
|
||||
- Allows inspection/modification for reproducibility
|
||||
- Enables dependency analysis before execution
|
||||
|
||||
**Why scoped modules?**
|
||||
- Organization namespacing prevents name collisions (`nf-core/salmon` vs `myorg/salmon`)
|
||||
- Clear ownership and provenance of modules
|
||||
- Supports private registries per scope
|
||||
- Industry-standard pattern (NPM, Terraform, others)
|
||||
- Enables ecosystem organization by maintainer/organization
|
||||
|
||||
**Why semantic versioning?**
|
||||
- Clear compatibility guarantees
|
||||
- Industry standard (npm, cargo, Go modules)
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**:
|
||||
- Enables ecosystem-wide code reuse
|
||||
- Reproducible workflows via committing the `modules/` directory (including `meta.yml`) to the project git repository
|
||||
- Centralized discovery and distribution via unified registry
|
||||
- Minimal operational overhead (single registry for both plugins and modules)
|
||||
- Module scoping enables organization namespaces and private registries
|
||||
- Local `modules/` directory provides project isolation
|
||||
- No version duplication: installed `meta.yml` is the single source of truth
|
||||
- Simple module structure: each module has single `main.nf` entry point
|
||||
|
||||
**Negative**:
|
||||
- Registry becomes critical infrastructure (requires HA setup)
|
||||
- Type-specific handling adds registry complexity
|
||||
- Parse-time resolution adds latency to workflow startup
|
||||
- Local `modules/` directory duplicates storage across projects (unlike global cache)
|
||||
|
||||
**Neutral**:
|
||||
- Modules and plugins conceptually distinct but share infrastructure
|
||||
- Different resolution timing supported by same API
|
||||
|
||||
## Links
|
||||
|
||||
- Related: [Plugin Spec ADR](20250922-plugin-spec.md)
|
||||
- Inspired by: [Go Modules](https://go.dev/ref/mod), [npm](https://docs.npmjs.com), [Cargo](https://doc.rust-lang.org/cargo/)
|
||||
- Related: [nf-core modules](https://nf-co.re/modules)
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: Module Schema Specification
|
||||
|
||||
This appendix defines the JSON schema for module `meta.yml` files. The schema maintains backward compatibility with existing nf-core module metadata patterns while supporting the new Nextflow module system features.
|
||||
|
||||
**Schema File:** [module-spec-schema.json](module-spec-schema.json)
|
||||
**Published URL:** `https://registry.nextflow.io/schemas/module-spec/v1.0.0`
|
||||
|
||||
### Field Reference
|
||||
|
||||
#### Core Fields (Existing nf-core Pattern)
|
||||
|
||||
These fields are already widely adopted in the nf-core community and remain fully supported:
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `name` | string | Yes | Module identifier |
|
||||
| `description` | string | Yes | Brief description of module functionality |
|
||||
| `keywords` | array[string] | Recommended | Discovery and categorization keywords |
|
||||
| `authors` | array[string] | Recommended | Original authors (GitHub handles) |
|
||||
| `maintainers` | array[string] | Recommended | Current maintainers |
|
||||
| `tools` | array[object] | Conditional | Software tools wrapped by the module |
|
||||
| `input` | array/object | Recommended | Input channel specifications |
|
||||
| `output` | object/array | Recommended | Output channel specifications |
|
||||
|
||||
#### Extension Fields (Nextflow Module System)
|
||||
|
||||
These fields extend the schema to support the new Nextflow module system:
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `version` | string | Registry | Semantic version (MAJOR.MINOR.PATCH) |
|
||||
| `license` | string | Registry | SPDX license identifier for module code |
|
||||
| `requires` | object | Optional | Runtime requirements |
|
||||
| `requires.nextflow` | string | Optional | Nextflow version constraint |
|
||||
|
||||
### Detailed Field Specifications
|
||||
|
||||
#### `name`
|
||||
|
||||
The module name must be a fully qualified scoped identifier in `scope/name` format:
|
||||
|
||||
```yaml
|
||||
name: nf-core/fastqc
|
||||
name: nf-core/bwa-mem
|
||||
name: myorg/custom-aligner
|
||||
```
|
||||
|
||||
**Naming Rules:**
|
||||
- Format: `scope/name` (e.g., `nf-core/salmon`, `myorg/custom`)
|
||||
- Scope: lowercase alphanumeric with hyphens (organization/owner identifier)
|
||||
- Name: lowercase alphanumeric with underscores/hyphens (module identifier)
|
||||
- Pattern: `^[a-z0-9][a-z0-9-]*/[a-z][a-z0-9_-]*$`
|
||||
|
||||
#### `version`
|
||||
|
||||
Semantic version following [SemVer 2.0.0](https://semver.org/):
|
||||
|
||||
```yaml
|
||||
version: "1.0.0"
|
||||
version: "2.3.1"
|
||||
version: "1.0.0-beta.1"
|
||||
```
|
||||
|
||||
**Version Semantics:**
|
||||
- **MAJOR:** Breaking changes to process signatures, inputs, or outputs
|
||||
- **MINOR:** New processes, backward-compatible enhancements
|
||||
- **PATCH:** Bug fixes, documentation updates
|
||||
|
||||
**Requirement:** Mandatory for registry-published modules (scoped names in `scope/name` format).
|
||||
|
||||
#### `requires`
|
||||
|
||||
Specifies runtime requirements for the module.
|
||||
|
||||
```yaml
|
||||
requires:
|
||||
nextflow: ">=24.04.0"
|
||||
```
|
||||
|
||||
**`requires.nextflow`** - Nextflow version constraint:
|
||||
```yaml
|
||||
requires:
|
||||
nextflow: ">=24.04.0" # minimum version
|
||||
nextflow: ">=24.04.0,<25.0.0" # version range
|
||||
```
|
||||
|
||||
#### `tools`
|
||||
|
||||
Documents the software tools wrapped by the module:
|
||||
|
||||
```yaml
|
||||
tools:
|
||||
- bwa:
|
||||
description: BWA aligner
|
||||
homepage: http://bio-bwa.sourceforge.net/
|
||||
license: ["GPL-3.0-or-later"]
|
||||
identifier: biotools:bwa
|
||||
```
|
||||
|
||||
**Tool Properties:**
|
||||
|
||||
| Property | Required | Description |
|
||||
|----------|----------|-------------|
|
||||
| `description` | Yes | Tool description |
|
||||
| `homepage` | One of these | Tool homepage URL |
|
||||
| `documentation` | One of these | Documentation URL |
|
||||
| `tool_dev_url` | One of these | Development/source URL |
|
||||
| `doi` | One of these | Publication DOI |
|
||||
| `arxiv` | No | arXiv identifier |
|
||||
| `license` | Recommended | SPDX license(s) |
|
||||
| `identifier` | Recommended | bio.tools identifier |
|
||||
| `manual` | No | User manual URL |
|
||||
|
||||
#### `input` and `output`
|
||||
|
||||
The schema supports a simplified syntax for inputs and outputs. Tooling such as the `module` CLI and module registry supports both the module spec and nf-core syntax for backwards compatibility:
|
||||
|
||||
**Module Spec:**
|
||||
```yaml
|
||||
input:
|
||||
- - name: meta
|
||||
type: map
|
||||
description: Sample metadata
|
||||
- name: reads
|
||||
type: file
|
||||
description: Input FastQ files
|
||||
ontologies:
|
||||
- edam: "http://edamontology.org/format_1930"
|
||||
- name: index
|
||||
type: directory
|
||||
description: Reference index
|
||||
|
||||
output:
|
||||
- - name: meta
|
||||
type: map
|
||||
description: Sample metadata
|
||||
- name: bam
|
||||
type: file
|
||||
description: Aligned BAM file
|
||||
pattern: "*.bam"
|
||||
- name: versions
|
||||
type: file
|
||||
description: Software versions
|
||||
```
|
||||
|
||||
**nf-core module (legacy):**
|
||||
```yaml
|
||||
input:
|
||||
- - meta:
|
||||
type: map
|
||||
description: Sample metadata
|
||||
- reads:
|
||||
type: file
|
||||
description: Input FastQ files
|
||||
ontologies:
|
||||
- edam: "http://edamontology.org/format_1930"
|
||||
- - index:
|
||||
type: directory
|
||||
description: Reference index
|
||||
|
||||
output:
|
||||
bam:
|
||||
- - meta:
|
||||
type: map
|
||||
description: Sample metadata
|
||||
- "*.bam":
|
||||
type: file
|
||||
description: Aligned BAM file
|
||||
pattern: "*.bam"
|
||||
versions:
|
||||
- versions.yml:
|
||||
type: file
|
||||
description: Software versions
|
||||
```
|
||||
|
||||
**Module Input/Output Properties:**
|
||||
|
||||
| Property | Type | Description |
|
||||
|----------|------|-------------|
|
||||
| `type` | string | Data type: `map`, `file`, `directory`, `string`, `integer`, `float`, `boolean`, `list`, `val` |
|
||||
| `description` | string | Human-readable description |
|
||||
| `pattern` | string | File glob pattern or value pattern |
|
||||
| `optional` | boolean | Whether input is optional (default: false) |
|
||||
| `default` | any | Default value if not provided |
|
||||
| `enum` | array | List of allowed values |
|
||||
| `ontologies` | array | EDAM or other ontology annotations |
|
||||
|
||||
### Migration Guide
|
||||
|
||||
#### From nf-core Module to Registry Module
|
||||
|
||||
**Before (nf-core local):**
|
||||
```yaml
|
||||
name: bwa_mem
|
||||
description: Align reads using BWA-MEM
|
||||
keywords:
|
||||
- alignment
|
||||
- bwa
|
||||
tools:
|
||||
- bwa:
|
||||
description: BWA software
|
||||
homepage: http://bio-bwa.sourceforge.net/
|
||||
license: ["GPL-3.0-or-later"]
|
||||
identifier: biotools:bwa
|
||||
authors:
|
||||
- "@drpatelh"
|
||||
maintainers:
|
||||
- "@drpatelh"
|
||||
input:
|
||||
# ... existing input spec
|
||||
output:
|
||||
# ... existing output spec
|
||||
```
|
||||
|
||||
**After (Registry-ready):**
|
||||
```yaml
|
||||
name: nf-core/bwa-mem # Added scope prefix
|
||||
version: "1.0.0" # Added version
|
||||
description: Align reads using BWA-MEM
|
||||
keywords:
|
||||
- alignment
|
||||
- bwa
|
||||
license: MIT # Added module license
|
||||
requires: # Added requirements
|
||||
nextflow: ">=24.04.0"
|
||||
tools:
|
||||
- bwa:
|
||||
description: BWA software
|
||||
homepage: http://bio-bwa.sourceforge.net/
|
||||
license: ["GPL-3.0-or-later"]
|
||||
identifier: biotools:bwa
|
||||
authors:
|
||||
- "@drpatelh"
|
||||
maintainers:
|
||||
- "@drpatelh"
|
||||
input:
|
||||
# ... modified as shown above
|
||||
output:
|
||||
# ... modified as shown above
|
||||
```
|
||||
|
||||
#### Schema Validation
|
||||
|
||||
Use the schema reference in your `meta.yml`:
|
||||
|
||||
```yaml
|
||||
# yaml-language-server: $schema=https://registry.nextflow.io/schemas/module-spec/v1.0.0
|
||||
|
||||
name: nf-core/my-module
|
||||
version: "1.0.0"
|
||||
# ...
|
||||
```
|
||||
|
||||
### Compatibility Matrix
|
||||
|
||||
| Feature | nf-core Current | Nextflow Module System |
|
||||
|---------|-----------------|------------------------|
|
||||
| Simple names | Yes | Yes (local only) |
|
||||
| Scoped names | No | Yes (registry) |
|
||||
| Version field | No | Yes (required for registry) |
|
||||
| `tools` section | Yes | Yes |
|
||||
| `components` | Yes | Deprecated |
|
||||
| `requires` | No | Yes (Nextflow version constraint) |
|
||||
| I/O specifications | Yes | Yes |
|
||||
| Ontologies | Yes | Yes |
|
||||
|
||||
### Unsupported nf-core Attributes
|
||||
|
||||
The following attributes from the nf-core meta schema are **not supported** in the Nextflow module system:
|
||||
|
||||
| Attribute | Reason | Alternative |
|
||||
|-----------|--------|-------------|
|
||||
| `extra_args` | Not adopted in practice by nf-core modules | To be defined |
|
||||
| `components` | No longer supported | Module dependencies are managed via `nextflow.config` |
|
||||
|
||||
### Complete Examples
|
||||
|
||||
#### Minimal nf-core Module
|
||||
|
||||
```yaml
|
||||
name: fastqc
|
||||
description: Run FastQC on sequenced reads
|
||||
keywords:
|
||||
- quality control
|
||||
- qc
|
||||
- fastq
|
||||
tools:
|
||||
- fastqc:
|
||||
description: FastQC quality metrics
|
||||
homepage: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/
|
||||
license: ["GPL-2.0-only"]
|
||||
identifier: biotools:fastqc
|
||||
authors:
|
||||
- "@drpatelh"
|
||||
maintainers:
|
||||
- "@drpatelh"
|
||||
input:
|
||||
- - meta:
|
||||
type: map
|
||||
description: Metadata map
|
||||
- reads:
|
||||
type: file
|
||||
description: List of input FastQ files
|
||||
output:
|
||||
html:
|
||||
- "*.html":
|
||||
type: file
|
||||
description: FastQC HTML report
|
||||
versions:
|
||||
- versions.yml:
|
||||
type: file
|
||||
description: Software versions
|
||||
```
|
||||
|
||||
#### Full Registry Module
|
||||
|
||||
```yaml
|
||||
name: nf-core/bwa-align
|
||||
version: "1.2.4"
|
||||
description: Align reads to reference genome using BWA-MEM algorithm
|
||||
keywords:
|
||||
- alignment
|
||||
- mapping
|
||||
- bwa
|
||||
- bam
|
||||
- fastq
|
||||
license: MIT
|
||||
|
||||
requires:
|
||||
nextflow: ">=24.04.0"
|
||||
|
||||
tools:
|
||||
- bwa:
|
||||
description: |
|
||||
BWA is a software package for mapping DNA sequences
|
||||
against a large reference genome.
|
||||
homepage: http://bio-bwa.sourceforge.net/
|
||||
documentation: https://bio-bwa.sourceforge.net/bwa.shtml
|
||||
doi: 10.1093/bioinformatics/btp324
|
||||
license: ["GPL-3.0-or-later"]
|
||||
identifier: biotools:bwa
|
||||
|
||||
authors:
|
||||
- "@nf-core"
|
||||
maintainers:
|
||||
- "@drpatelh"
|
||||
- "@maxulysse"
|
||||
|
||||
input:
|
||||
- - name: meta
|
||||
type: map
|
||||
description: Sample metadata map (e.g., [ id:'sample1', single_end:false ])
|
||||
- name: reads
|
||||
type: file
|
||||
description: Input FastQ files
|
||||
ontologies:
|
||||
- edam: "http://edamontology.org/format_1930"
|
||||
- - name: meta2
|
||||
type: map
|
||||
description: Reference metadata
|
||||
- name: index
|
||||
type: directory
|
||||
description: BWA index directory
|
||||
ontologies:
|
||||
- edam: "http://edamontology.org/data_3210"
|
||||
|
||||
output:
|
||||
- - name: meta
|
||||
type: map
|
||||
description: Sample metadata
|
||||
- name: bam
|
||||
type: file
|
||||
description: Aligned BAM file
|
||||
pattern: "*.bam"
|
||||
ontologies:
|
||||
- edam: "http://edamontology.org/format_2572"
|
||||
- name: versions
|
||||
type: file
|
||||
description: Software versions
|
||||
pattern: "versions.yml"
|
||||
```
|
||||
|
||||
252
nextflow/adr/20251205-git-multi-revision-asset-management.md
Normal file
252
nextflow/adr/20251205-git-multi-revision-asset-management.md
Normal file
@@ -0,0 +1,252 @@
|
||||
# Git Multi-Revision Asset Management with Strategy Pattern
|
||||
|
||||
- Authors: Jorge Ejarque
|
||||
- Status: Approved
|
||||
- Deciders: Jorge Ejarque, Ben Sherman, Paolo Di Tommaso
|
||||
- Date: 2025-12-05
|
||||
- Tags: scm, asset-management, multi-revision
|
||||
|
||||
## Summary
|
||||
|
||||
Nextflow's asset management system has been refactored to support multiple revisions of the same pipeline concurrently through a bare repository approach with shared object storage, while maintaining backward compatibility with legacy direct-clone repositories using the Strategy design pattern.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The original asset management system (`AssetManager`) cloned each pipeline directly to `~/.nextflow/assets/<org>/<project>/.git`, creating several limitations:
|
||||
|
||||
1. **No concurrent Git multi-revision support**: Only one revision of a pipeline could be checked out at a time, preventing concurrent execution of different versions
|
||||
2. **Update conflicts**: Pulling updates while a pipeline was running could cause conflicts or corruption
|
||||
3. **Testing limitations**: Users couldn't easily test different versions of a pipeline side-by-side
|
||||
|
||||
The goal was to enable running multiple revisions of the same pipeline concurrently (e.g., production on v1.0, testing on v2.0-dev) while maintaining efficient disk usage through object sharing.
|
||||
|
||||
## Goals or Decision Drivers
|
||||
|
||||
- **Concurrent multi-revision execution**: Must support running different revisions of the same pipeline simultaneously
|
||||
- **Efficient disk usage**: Share Git objects between revisions to minimize storage overhead
|
||||
- **Backward compatibility**: Must not break existing pipelines using the legacy direct-clone approach
|
||||
- **API stability**: Maintain the existing `AssetManager` API for external consumers (K8s plugin, CLI commands, etc.)
|
||||
- **Minimal migration impact**: Existing repositories should continue to work without user intervention
|
||||
- **JGit compatibility**: Solution must work within JGit's capabilities to avoid relying on Git client installations
|
||||
- **Atomic updates**: Downloading new revisions should not interfere with running pipelines
|
||||
|
||||
## Non-goals
|
||||
|
||||
- **Migration of existing legacy repositories**: Legacy repos continue to work as-is; no forced migration
|
||||
- **Native Git worktree support**: Due to JGit limitations, not using Git's worktree feature
|
||||
- **Revision garbage collection**: No automatic cleanup of old revisions (users can manually drop)
|
||||
- **Multi-hub support**: Still tied to a single repository provider per pipeline
|
||||
|
||||
## Considered Options
|
||||
|
||||
### Option 1: Bare Repository with Git Worktrees
|
||||
|
||||
Use Git's worktree feature to create multiple working directories from a single bare repository.
|
||||
|
||||
**Implementation**:
|
||||
- One bare repository at `~/.nextflow/assets/<org>/<project>/.git`
|
||||
- Multiple worktrees at `~/.nextflow/assets/<org>/<project>/<revision>/`
|
||||
|
||||
- Good, because it's the native Git solution for multiple checkouts
|
||||
- Good, because worktrees are space-efficient
|
||||
- Good, because Git handles all the complexity
|
||||
- **Bad, because JGit doesn't support worktrees** (deal-breaker)
|
||||
- Bad, because requires native Git installation
|
||||
|
||||
**Decision**: Rejected due to JGit incompatibility
|
||||
|
||||
### Option 2: Bare Repository + Clones per Commit + Revision Map File
|
||||
|
||||
Use a bare repository for storage and create clones for each commit, tracking them in a separate file.
|
||||
|
||||
**Implementation**:
|
||||
- Bare repository at `~/.nextflow/assets/<org>/<project>/.nextflow/bare_repo/`
|
||||
- Clones at `~/.nextflow/assets/<org>/<project>/.nextflow/commits/<commit-sha>/`
|
||||
- Revision map file at `~/.nextflow/assets/<org>/<project>/.nextflow/revisions.json` mapping revision names to commit SHAs
|
||||
|
||||
- Good, because it works with JGit
|
||||
- Good, because bare repo reduces remote repository interactions to checkout commits
|
||||
- Good, because explicit revision tracking
|
||||
- Bad, because disk space as git objects replicated in clones
|
||||
- Bad, because revision map file can become stale
|
||||
- Bad, because requires file I/O for every revision lookup
|
||||
- Bad, because potential race conditions on map file updates
|
||||
- Bad, because adds complexity of maintaining external state
|
||||
|
||||
**Decision**: Initially implemented but later refined
|
||||
|
||||
### Option 3: Bare Repository + Shared Clones with Strategy Pattern
|
||||
|
||||
Similar to Option 2 but eliminate the separate revision map file by using the bare repository itself as the source of truth. Additionally, use the Strategy pattern to maintain backward compatibility with existing legacy repositories without requiring migration.
|
||||
|
||||
**Implementation**:
|
||||
- Bare repository at `~/.nextflow/assets/.repos/<org>/<project>/bare/`
|
||||
- Shared clones at `~/.nextflow/assets/.repos/<org>/<project>/clones/<commit-sha>/`
|
||||
- Use bare repository refs to resolve revisions to commit SHAs dynamically
|
||||
- JGit alternates mechanism for object sharing
|
||||
- `AssetManager` as facade with unchanged public API
|
||||
- `RepositoryStrategy` interface defining repository operations
|
||||
- `LegacyRepositoryStrategy` for existing direct-clone behavior
|
||||
- `MultiRevisionRepositoryStrategy` for new bare-repo approach
|
||||
- Strategy selection based on environment variable or repository state detection
|
||||
|
||||
- Good, because no external state file to maintain
|
||||
- Good, because bare repository is always in sync (fetched on updates)
|
||||
- Good, because simpler and more reliable
|
||||
- Good, because atomic updates (Git operations are atomic)
|
||||
- Good, because works entirely within JGit
|
||||
- Good, because zero migration needed for existing repositories
|
||||
- Good, because maintains API compatibility
|
||||
- Good, because allows gradual adoption
|
||||
- Good, because isolates legacy code
|
||||
- Good, because makes future strategies easy to add
|
||||
- Neutral, because adds abstraction layer
|
||||
- Bad, because requires resolution on every access (minimal overhead)
|
||||
- Bad, because increases codebase size initially
|
||||
|
||||
**Decision**: Selected
|
||||
|
||||
## Solution or decision outcome
|
||||
|
||||
Implemented **Option 3 (Bare Repository + Shared Clones with Strategy Pattern)** for multi-revision support with backward compatibility. Multi-revision is the default for new repositories, while legacy mode is available via `NXF_SCM_LEGACY` environment variable.
|
||||
|
||||
## Rationale & discussion
|
||||
|
||||
### Git Multi-Revision Implementation
|
||||
|
||||
The bare repository approach provides efficient multi-revision support:
|
||||
|
||||
```
|
||||
~/.nextflow/assets/.repos/nextflow-io/hello/
|
||||
├── bare/ # Bare repository (shared objects)
|
||||
│ ├── objects/ # All Git objects stored here
|
||||
│ ├── refs/
|
||||
│ │ ├── heads/
|
||||
│ │ └── tags/
|
||||
│ └── config
|
||||
│
|
||||
└── clones/ # Revisions-specific clones
|
||||
├── abc123.../ # Clone for commit abc123
|
||||
│ └── .git/
|
||||
│ ├── objects/ # (uses alternates → bare/objects)
|
||||
│ └── info/
|
||||
│ └── alternates # Points to bare/objects
|
||||
│
|
||||
└── def456.../ # Clone for commit def456
|
||||
└── .git/
|
||||
|
||||
~/.nextflow/assets/nextflow-io/hello/
|
||||
└── .git/ # Legacy repo location (HYBRID state)
|
||||
```
|
||||
|
||||
**Key mechanisms:**
|
||||
|
||||
1. **Bare repository as source of truth**: The bare repo is fetched/updated from the remote, keeping refs current
|
||||
2. **Dynamic resolution**: Revisions (branch/tag names) are resolved to commit SHAs using the bare repo's refs
|
||||
3. **Object sharing**: Clones use Git alternates to reference the bare repo's objects, avoiding duplication
|
||||
4. **Atomic operations**: Each clone is independent; downloading a new revision doesn't affect existing ones
|
||||
5. **Lazy creation**: Clones are created on-demand when a specific revision is requested
|
||||
|
||||
**Advantages over revision map file:**
|
||||
- No external state to maintain or keep in sync
|
||||
- Bare repo fetch automatically updates all refs
|
||||
- Resolution is simple: `bareRepo.resolve(revision)` returns commit SHA
|
||||
- No race conditions on file updates
|
||||
- Simpler code with fewer failure modes
|
||||
|
||||
### Strategy Pattern for Backward Compatibility
|
||||
|
||||
The Strategy pattern provides clean separation and backward compatibility:
|
||||
|
||||
```
|
||||
┌─────────────────────────┐
|
||||
│ AssetManager │ ← Public API (unchanged)
|
||||
│ (Facade) │
|
||||
└───────────┬─────────────┘
|
||||
│
|
||||
│ delegates to
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ RepositoryStrategy │ ← Interface
|
||||
└───────────┬─────────────┘
|
||||
△
|
||||
│ implements
|
||||
┌───────┴────────┐
|
||||
│ │
|
||||
┌───────────┐ ┌─────────────────┐
|
||||
│ Legacy │ │ MultiRevision │ ← Concrete strategies
|
||||
│ Strategy │ │ Strategy │
|
||||
└───────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
**Strategy selection logic:**
|
||||
|
||||
1. Check `NXF_SCM_LEGACY` environment variable → Use legacy if set
|
||||
2. Check if there is only the legacy asset of the repository (`isOnlyLegacy` method) → Use legacy (preserve existing)
|
||||
3. Otherwise -> Use multi-revision
|
||||
|
||||
|
||||
**Backward compatibility guarantees:**
|
||||
|
||||
- Existing repositories continue to work without changes
|
||||
- `AssetManager` API remains identical
|
||||
- CLI commands work with both strategies transparently
|
||||
- Tests pass with minimal modifications
|
||||
- No forced migration; users opt-in naturally when creating new repos
|
||||
|
||||
### Hybrid State Handling
|
||||
|
||||
The system gracefully handles hybrid states where both legacy and multi-revision repositories coexist:
|
||||
|
||||
- **Detection**: In hybrid states, a multi-revision strategy is selected by default.
|
||||
- **Fallback logic**: Multi-revision strategy can fall back to legacy repo for operations if needed
|
||||
- **No conflicts**: Strategies are designed to coexist; operations target different directories
|
||||
- **Explicit control**: Users can force a specific strategy via `setStrategyType()` or `NXF_SCM_LEGACY` environment variable
|
||||
|
||||
### Migration Path
|
||||
|
||||
Users naturally migrate as they pull new revisions:
|
||||
|
||||
1. **Existing users**: Can continue with legacy repos (`NXF_SCM_LEGACY` state detected)
|
||||
2. **New users**: Get multi-revision by default
|
||||
3. **Opt-in migration**: Delete project directory to switch to multi-revision or pull with --migrate
|
||||
4. **Opt-out**: Set `NXF_SCM_LEGACY=true` to force legacy mode
|
||||
|
||||
### Implementation Details
|
||||
|
||||
**Key classes:**
|
||||
|
||||
- `RepositoryStrategy`: Interface defining repository operations
|
||||
- `AbstractRepositoryStrategy`: Base class with shared helper methods
|
||||
- `LegacyRepositoryStrategy`: Direct clone implementation (original behavior)
|
||||
- `MultiRevisionRepositoryStrategy`: Bare repo + shared clones implementation
|
||||
|
||||
**Critical methods:**
|
||||
|
||||
- `download()`: Equivalent for both strategies (legacy pulls, multi-revision creates shared clone)
|
||||
- `getLocalPath()`: Returns appropriate working directory based on strategy
|
||||
- `getGit()`: Returns appropriate Git instance (legacy git, bare git, or commit git)
|
||||
|
||||
### Performance Characteristics
|
||||
|
||||
**Disk usage:**
|
||||
- Legacy: ~100% per repository (full clone with all git objects) + Worktree
|
||||
- Multi-revision: ~100% for bare + ~100K (.git with alternates) per revision + Worktree per revision
|
||||
|
||||
**Operation speed:**
|
||||
- First download: Similar (both clone from remote)
|
||||
- Additional revisions: Multi-revision faster (only fetches new objects once, creates cheap clones)
|
||||
- Switching revisions: Multi-revision instant (different directories), legacy requires checkout
|
||||
|
||||
### Known Limitations
|
||||
|
||||
- No automatic migration of legacy repositories
|
||||
- Bare repository overhead even for users who only need one revision
|
||||
- JGit alternates slightly more complex than worktrees
|
||||
- Manual cleanup required for old revision clones
|
||||
|
||||
## Links
|
||||
- [GitHub Issue #2870 - Multiple revisions of the same pipeline for concurrent execution](https://github.com/nextflow-io/nextflow/issues/2870)
|
||||
- [PR #6620 - Implementation of multiple revisions without revisions map](https://github.com/nextflow-io/nextflow/pull/6620)
|
||||
- Related PRs implementing the multi-revision approach (linked in #6620)
|
||||
|
||||
484
nextflow/adr/20260306-record-types.md
Normal file
484
nextflow/adr/20260306-record-types.md
Normal file
@@ -0,0 +1,484 @@
|
||||
# Record types
|
||||
|
||||
- Authors: Ben Sherman
|
||||
- Status: accepted
|
||||
- Deciders: Ben Sherman, Paolo Di Tommaso
|
||||
- Date: 2026-03-06
|
||||
- Tags: lang, static-types
|
||||
|
||||
## Updates
|
||||
|
||||
### Version 1.1 (2026-03-23)
|
||||
|
||||
- Replaced inline record type syntax (`Record { ... }`) with destructuring syntax (`record(...)`) for better continuity with legacy syntax and record output syntax.
|
||||
|
||||
## Summary
|
||||
|
||||
Provide a way to model composite data types in the Nextflow language.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Nextflow pipelines typically need a way to model *aggregates*, or "data that travels together", such as a paired-end read consisting of a sample ID and two FASTQ files.
|
||||
|
||||
Primary use cases:
|
||||
|
||||
- Model complex pipeline inputs and outputs (e.g. samplesheets as collections of records)
|
||||
|
||||
- Generate JSON schemas for pipeline inputs and outputs from source code (e.g. to facilitate pipeline chaining with external validation)
|
||||
|
||||
- Model directory outputs as records, enabling more fine-grained validation
|
||||
|
||||
## Goals
|
||||
|
||||
- Introduce a alternative data structure to tuples that allows users to model their data domain more precisely.
|
||||
|
||||
- Introduce a way to validate custom data types at compile-time while keeping pipeline code concise and readable.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Removing support for tuples -- tuples should continue to work as before
|
||||
|
||||
- Introducing new dataflow operators or changing existing ones -- any changes to operators will be addressed in future efforts as needed
|
||||
|
||||
- Introducing type inheritance -- the Nextflow type system avoids type inheritance as much as possible in order to not introduce unnecessary complexity
|
||||
|
||||
- Introducing object methods -- this can be handled by standalone functions for now, and may be improved in a future effort (e.g. namespaces)
|
||||
|
||||
## Decision
|
||||
|
||||
Add support for **records**, an alternative data structure to tuples, and **record types**, a way to declare custom types that can be applied to records a la carte.
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
### Records
|
||||
|
||||
Records are an attempt to take the best qualities of maps and custom classes while avoiding the downsides.
|
||||
|
||||
A record can be created using the `record()` function:
|
||||
|
||||
```groovy
|
||||
sample = record(
|
||||
id: '1',
|
||||
fastq_1: file('1_1.fastq'),
|
||||
fastq_2: file('1_2.fastq')
|
||||
)
|
||||
|
||||
sample.id = '2' // error: record cannot be modified
|
||||
sample += record(id: '2') // ok
|
||||
|
||||
println sample.id
|
||||
```
|
||||
|
||||
This function effectively creates an immutable map (`Map<String,?>`):
|
||||
|
||||
- The keys are just field names
|
||||
- The values can have any type
|
||||
- The record can't be modified -- use the `+` operator instead
|
||||
|
||||
Records can have arbitrary fields, unlike custom classes, which makes them easy to use with dataflow operators.
|
||||
|
||||
For example, a future version of the `join` operator could join records as follows:
|
||||
|
||||
```groovy
|
||||
ch_bam = channel.of( record(id: '1', bam: file('1.bam')) )
|
||||
ch_bai = channel.of( record(id: '1', bai: file('1.bai')) )
|
||||
|
||||
ch_bam.join(ch_bai, by: 'id').view()
|
||||
|
||||
// -> record(id: '1', bam: file('1.bam'), bai: file('1.bai'))
|
||||
```
|
||||
|
||||
Whereas tuples are joined by a matching index, records would be joined by a matching key.
|
||||
|
||||
### Record types
|
||||
|
||||
A record type is a user-defined type that consists of a name and a set of fields:
|
||||
|
||||
```groovy
|
||||
record Sample {
|
||||
id: String
|
||||
fastq_1: Path
|
||||
fastq_2: Path?
|
||||
}
|
||||
```
|
||||
|
||||
Fields in a record type are declared the same way as [typed parameters](https://nextflow.io/docs/latest/workflow.html#typed-parameters). All [standard types](https://nextflow.io/docs/latest/reference/stdlib-types.html) can be used. Fields can be marked as optional by appending a `?` to the field type.
|
||||
|
||||
The purpose of a record type is to specify a *minimum set of requirements* for a record *in a particular context*. A record created with the `record()` function simply has the type `Record` -- it makes no guarantees about which fields it provides. A *record type* can be used (e.g. in a workflow input) to make a stronger guarantee.
|
||||
|
||||
For example:
|
||||
|
||||
```groovy
|
||||
workflow RNASEQ {
|
||||
take:
|
||||
samples: Channel<Sample>
|
||||
|
||||
main:
|
||||
// ...
|
||||
}
|
||||
|
||||
workflow {
|
||||
ch_samples = channel.of( record(id: '1', fastq_2: file('1_2.fastq')) )
|
||||
RNASEQ(ch_samples) // error: `ch_samples` is missing `fastq_1` field required by Sample
|
||||
}
|
||||
```
|
||||
|
||||
This workflow definition specifies that the `samples` input should be a channel of records, where each record has at minimum the fields specified by the `Sample` record type. The records can still have additional fields, but only the fields in `Sample` are guaranteed to be present.
|
||||
|
||||
In other words, records are *duck-typed*. Duck-typing semantics are used whenever a record is validated against a record type:
|
||||
|
||||
- Supplying a record argument to a workflow, process, or function with a record type input (as shown above)
|
||||
|
||||
- Casting a record to a record type (e.g. `record(...) as Sample`)
|
||||
|
||||
Record types can be included across modules:
|
||||
|
||||
```groovy
|
||||
include { Sample } from './module'
|
||||
```
|
||||
|
||||
Because of duck-typing, two record types with the same fields and field types are effectively equivalent, even if they have different names:
|
||||
|
||||
```groovy
|
||||
record Sample {
|
||||
id: String
|
||||
fastq_1: Path
|
||||
fastq_2: Path?
|
||||
}
|
||||
|
||||
record FastqPair {
|
||||
id: String
|
||||
fastq_1: Path
|
||||
fastq_2: Path?
|
||||
}
|
||||
```
|
||||
|
||||
This makes it easier to compose modules and workflows that use their own record types.
|
||||
|
||||
### Process inputs
|
||||
|
||||
When a record is supplied as input to a process, the process needs to know how to stage input files from the record, like it does with the `path` qualifier.
|
||||
|
||||
Typed processes can stage inputs using the `stage:` section, but ideally the files in a record should be automatically detected and staged.
|
||||
|
||||
A typed process can declare a record input using a record type:
|
||||
|
||||
```groovy
|
||||
process FASTQC {
|
||||
input:
|
||||
sample: FastqPair
|
||||
|
||||
// ...
|
||||
}
|
||||
|
||||
record FastqPair {
|
||||
id: String
|
||||
fastq_1: Path
|
||||
fastq_2: Path
|
||||
}
|
||||
```
|
||||
|
||||
All record fields that are a `Path` or `Path` collection (e.g. `Set<Path>`) are automatically staged. The record itself is declared in the process body as `sample`, like any other input, and record fields are accessed as `sample.id`, `sample.fastq_1`, and so on.
|
||||
|
||||
Alternatively, a typed process can declare a *destructured* record input:
|
||||
|
||||
```groovy
|
||||
process FASTQC {
|
||||
input:
|
||||
record(
|
||||
id: String,
|
||||
fastq_1: Path,
|
||||
fastq_2: Path
|
||||
)
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
This approach allows record inputs to be declared without the need for external record types. Each record field is acessed directly as `id`, `fastq_1`, and so on.
|
||||
|
||||
### Process outputs
|
||||
|
||||
Typed processes can declare outputs with arbitrary expressions, so no new syntax is required to support record outputs. Simply use the `record()` function to create a record:
|
||||
|
||||
```groovy
|
||||
process FASTQC {
|
||||
// ...
|
||||
|
||||
output:
|
||||
record(
|
||||
id: id,
|
||||
fastqc: file('fastqc_logs')
|
||||
)
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
The type of this output is an *implicit* record type that is inferred from the code: `Record { id: String ; fastqc: Path }`.
|
||||
|
||||
## Alternatives
|
||||
|
||||
### Custom classes
|
||||
|
||||
Define Groovy-style classes (see #2085) and use them to model composite data:
|
||||
|
||||
```groovy
|
||||
@nextflow.io.ValueObject
|
||||
class Sample {
|
||||
String id
|
||||
Path fastq_1
|
||||
Path fastq_2
|
||||
}
|
||||
|
||||
workflow {
|
||||
sample = new Sample('1', file('1_1.fastq'), file('1_2.fastq'))
|
||||
|
||||
println sample.id
|
||||
}
|
||||
```
|
||||
|
||||
This approach can be used, but in practice it requires a lot of extra dataflow logic around process calls to convert between custom types and tuples, because processes don’t know how to stage input files from custom types.
|
||||
|
||||
Custom classes are not very flexible. For example, joining two channels of custom classes would be more complicated than joining two tuples by a matching key, because you would need to define an additional class for the “joined” type and explicitly construct it from the two joining classes.
|
||||
|
||||
### Maps
|
||||
|
||||
Use maps to create composite data structures dynamically (see #2127):
|
||||
|
||||
```groovy
|
||||
sample = [
|
||||
id: '1',
|
||||
fastq_1: file('1_1.fastq'),
|
||||
fastq_2: file('1_2.fastq')
|
||||
]
|
||||
|
||||
println sample.id
|
||||
```
|
||||
|
||||
Maps are flexible because you can store arbitrary fields rather than being restricted to a fixed set of fields. However, maps are meant to be used for a single value type (e.g. `Map<String,Integer>`).
|
||||
|
||||
Unlike tuples, maps are mutable (i.e. they can be modified). Modifying maps can lead to race conditions if done improperly. As a best practice, maps should be modified by adding another map, which creates a copy:
|
||||
|
||||
```groovy
|
||||
sample2 = sample + [id: '2']
|
||||
|
||||
println sample.id // -> '1'
|
||||
println sample2.id // -> '2'
|
||||
```
|
||||
|
||||
### Inline record input type
|
||||
|
||||
A process can declare a destructured record input as shown above:
|
||||
|
||||
```groovy
|
||||
process FASTQC {
|
||||
input:
|
||||
record(
|
||||
id: String,
|
||||
fastq_1: Path,
|
||||
fastq_2: Path
|
||||
)
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
One alternative is to declare an *inline record type*:
|
||||
|
||||
```groovy
|
||||
process FASTQC {
|
||||
input:
|
||||
sample: Record {
|
||||
id: String
|
||||
fastq_1: Path
|
||||
fastq_2: Path
|
||||
}
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
This approach was considered because it uses the same syntax as a `record` definition, making it easy to switch between inline and external record types. The block syntax is also slightly better suited for a type definition since it doesn't require commas.
|
||||
|
||||
However, this approach creates an asymmetry between record inputs and outputs (`Record { ... }` vs `record(...)`). It also removes the ability to destructure a record input.
|
||||
|
||||
Declaring a record input with `record()` can be understood as a reverse constructor, mirroring the `record()` function used to construct a record output in the `output:` section.
|
||||
|
||||
While both approaches have pros and cons, the `record()` approach was ultimately chosen for its continuity with the existing tuple syntax and its similarity with the record output syntax.
|
||||
|
||||
### Implicit process record output
|
||||
|
||||
A process record output can be defined using the `record()` function as shown above:
|
||||
|
||||
```groovy
|
||||
process FASTQC {
|
||||
// ...
|
||||
|
||||
output:
|
||||
record(
|
||||
id: id,
|
||||
fastqc: file('fastqc_logs')
|
||||
)
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
One alternative is to re-interpret the existing typed output syntax as an implicit record, treating each line as a record field:
|
||||
|
||||
```groovy
|
||||
process FASTQC {
|
||||
// ...
|
||||
|
||||
output:
|
||||
id: String = id
|
||||
fastqc: Path = file('fastqc_logs')
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
This approach is syntactically more concise, and it re-uses the typed output syntax that was introduced in Nextflow 25.10.
|
||||
|
||||
However, with this approach, the same syntax can have different meanings depending on the surrounding context (e.g. presence/absence of the `nextflow.enable.types` feature flag), which can be confusing for both users and agents.
|
||||
|
||||
The `record()` approach works "out of the box", and it isn't much more verbose, so we decided that it is sufficient for now.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive:**
|
||||
|
||||
- Records replace positional elements with named fields, making data structures self-documenting and less error-prone (e.g. no more accidentally swapping `fastq_1` and `fastq_2` by position).
|
||||
|
||||
- Duck-typing makes module composition easier: downstream processes declare only the fields they need, and record types from different modules are interchangeable if they share the same fields.
|
||||
|
||||
- Immutability by default eliminates the race conditions that can occur when mutable maps are improperly used in workflow logic.
|
||||
|
||||
- A single record output replaces multiple per-file tuple output channels (as shown in the prokka example), reducing the total number of channels in a workflow.
|
||||
|
||||
- Record types provide a foundation for generating JSON schemas for pipeline inputs and outputs directly from source code, enabling external validation and pipeline chaining.
|
||||
|
||||
- Backward compatibility is fully preserved: existing tuple-based pipelines continue to work without modification.
|
||||
|
||||
**Negative:**
|
||||
|
||||
- Since records must match based on field name rather than element input, users must be careful to use consistent naming conventions or write additional adaptor logic in their workflow. Type checking for records will be essential to streamline the developer experience as much as possible.
|
||||
|
||||
- Tuples and records coexist as parallel data model options, which may cause confusion about which to use for a given situation. Guidelines will be needed to help users make the right choice.
|
||||
|
||||
- Dataflow operators such as `cross`, `groupTuple`, and `join` need to be updated to support records natively.
|
||||
|
||||
**Neutral:**
|
||||
|
||||
- Record types use structural (duck) typing rather than nominal typing. Two record types with identical fields are interchangeable regardless of their names. This is intentional and enables flexible module composition, but it differs from the nominal typing that most users encounter in other languages, so it may be surprising at first.
|
||||
|
||||
- Records have no methods; behavior must be expressed via standalone functions. This is consistent with the functional style of Nextflow pipelines, and may be improved in the future with namespaces.
|
||||
|
||||
## Links
|
||||
|
||||
- Community issues: #2085, #2127
|
||||
- Related nf-core discussion: https://github.com/nf-core/modules/issues/4311
|
||||
- Original implementation: #4553
|
||||
- nf-core/fetchngs POC: https://github.com/nf-core/fetchngs/pull/309
|
||||
- Inspired by: [Simple Made Easy](https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/SimpleMadeEasy.md)
|
||||
- Type systems: [Nominal typing](https://en.wikipedia.org/wiki/Nominal_type_system) vs [Structural typing](https://en.wikipedia.org/wiki/Structural_type_system) vs [Duck typing](https://en.wikipedia.org/wiki/Duck_typing)
|
||||
|
||||
## Appendix
|
||||
|
||||
### Example: nf-core/prokka
|
||||
|
||||
The [nf-core/prokka](https://github.com/nf-core/modules/blob/master/modules/nf-core/prokka/main.nf) module produces several output files, emitting a tuple channel for each output file. If a downstream process requires multiple outputs, the individual output channels must be joined as needed to match the process input tuple.
|
||||
|
||||
Here is how the process might look using records:
|
||||
|
||||
```groovy
|
||||
process PROKKA {
|
||||
// ...
|
||||
|
||||
input:
|
||||
record(
|
||||
meta: Map,
|
||||
fasta: Path
|
||||
)
|
||||
proteins: Path
|
||||
prodigal_tf: Path
|
||||
|
||||
output:
|
||||
record(
|
||||
meta: meta,
|
||||
gff: file("${prefix}/*.gff"),
|
||||
gbk: file("${prefix}/*.gbk"),
|
||||
fna: file("${prefix}/*.fna"),
|
||||
faa: file("${prefix}/*.faa"),
|
||||
ffn: file("${prefix}/*.ffn"),
|
||||
sqn: file("${prefix}/*.sqn"),
|
||||
fsa: file("${prefix}/*.fsa"),
|
||||
tbl: file("${prefix}/*.tbl"),
|
||||
err: file("${prefix}/*.err"),
|
||||
log: file("${prefix}/*.log"),
|
||||
txt: file("${prefix}/*.txt"),
|
||||
tsv: file("${prefix}/*.tsv")
|
||||
)
|
||||
|
||||
topic:
|
||||
file("versions.yml") >> 'versions'
|
||||
|
||||
script:
|
||||
prefix = meta.id
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
The tuple input is refactored as a record input with an inline record type. The tuple outputs are combined into a single record output. No external record types are needed, although they could be used if desired.
|
||||
|
||||
*NOTE:* The `meta` map has not been changed in this example for brevity. However, it could be modeled with a record type instead of the generic `Map` type, or it could even be replaced with explicit fields such as `id: String`.
|
||||
|
||||
Now, suppose there are two downstream processes that want to use the outputs of `PROKKA`:
|
||||
|
||||
1. Process `FOO` only needs the `gff` file
|
||||
2. Process `BAR` only needs the `fna`, `faa`, and `tbl` files
|
||||
|
||||
These processes would be defined as follows:
|
||||
|
||||
```groovy
|
||||
process FOO {
|
||||
|
||||
input:
|
||||
record(
|
||||
meta: Map,
|
||||
gff: Path
|
||||
)
|
||||
|
||||
// ...
|
||||
}
|
||||
|
||||
process BAR {
|
||||
|
||||
input:
|
||||
record(
|
||||
meta: Map,
|
||||
fna: Path,
|
||||
faa: Path,
|
||||
tbl: Path
|
||||
)
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
And the calling workflow would be written as follows:
|
||||
|
||||
```groovy
|
||||
workflow {
|
||||
ch_inputs = channel.of( /* ... */ )
|
||||
proteins = // ...
|
||||
prodigal_tf = // ...
|
||||
cn_prokka = PROKKA( ch_inputs, proteins, prodigal_tf )
|
||||
|
||||
FOO(ch_prokka)
|
||||
BAR(ch_prokka)
|
||||
}
|
||||
```
|
||||
|
||||
Each process declares a record input containing only the fields that it needs. When the output of `PROKKA` is passed to `FOO` and `BAR`, each process stages only the files that it declared in the record input.
|
||||
136
nextflow/adr/20260310-seqera-dataset-filesystem.md
Normal file
136
nextflow/adr/20260310-seqera-dataset-filesystem.md
Normal file
@@ -0,0 +1,136 @@
|
||||
# NIO Filesystem for Seqera Platform Datasets
|
||||
|
||||
- Authors: Jorge Ejarque
|
||||
- Status: draft
|
||||
- Date: 2026-03-10
|
||||
- Tags: nio, filesystem, seqera, datasets, nf-tower
|
||||
|
||||
Technical Story: Enable Nextflow pipelines to read Seqera Platform datasets as ordinary file paths using `seqera://` URIs.
|
||||
|
||||
## Summary
|
||||
|
||||
Add a Java NIO `FileSystemProvider` to the `nf-tower` plugin that registers the `seqera://` scheme, allowing pipelines to reference Seqera Platform datasets (CSV/TSV) as standard file paths without manual download steps. The implementation reuses the existing `TowerClient` for all HTTP communication, inheriting authentication and retry behaviour.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Nextflow users managing datasets on the Seqera Platform must currently download dataset files manually or through custom scripts before referencing them in pipelines. There is no native integration between Nextflow's file abstraction and the Seqera Platform dataset API. This creates friction in workflows where datasets are the primary input and forces users to handle authentication, versioning, and file staging outside the pipeline definition.
|
||||
|
||||
## Goals or Decision Drivers
|
||||
|
||||
- Transparent access to Seqera Platform datasets using standard Nextflow file path syntax
|
||||
- Reuse of existing nf-tower plugin infrastructure (authentication, HTTP client, retry/backoff)
|
||||
- Hierarchical path browsing matching the platform's org/workspace/dataset structure
|
||||
- Extensible architecture that can support future Seqera-managed resource types (e.g. data-links)
|
||||
- No new plugin or module — feature lives within nf-tower
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Streaming large datasets — the Platform API does not support streaming; content is fully buffered on download
|
||||
- Implementing resource types beyond `datasets` — only the extensible architecture is required
|
||||
- Local caching across pipeline runs — Nextflow's standard task staging handles caching
|
||||
- Dataset management operations (delete, rename) — the filesystem is read-only in the initial implementation
|
||||
|
||||
## Considered Options
|
||||
|
||||
### Option 1: Standalone plugin with dedicated HTTP client
|
||||
|
||||
A new `nf-seqera-fs` plugin with its own HTTP client configuration and authentication setup.
|
||||
|
||||
- Good, because it isolates the filesystem code from the nf-tower plugin
|
||||
- Bad, because it duplicates authentication configuration and HTTP client setup
|
||||
- Bad, because two separate HTTP clients sharing a refresh token would corrupt each other's auth state
|
||||
|
||||
### Option 2: NIO filesystem within nf-tower using TowerClient delegation
|
||||
|
||||
Add the filesystem to nf-tower, delegating all HTTP through the existing `TowerClient` singleton via a typed `SeqeraDatasetClient` wrapper.
|
||||
|
||||
- Good, because it shares authentication and token refresh with TowerClient
|
||||
- Good, because it reuses existing retry/backoff configuration
|
||||
- Good, because no new dependencies are needed
|
||||
|
||||
### Option 3: Direct HxClient usage within nf-tower
|
||||
|
||||
Add the filesystem to nf-tower but use `HxClient` directly rather than going through TowerClient.
|
||||
|
||||
- Good, because it gives full control over request construction
|
||||
- Bad, because exposing HxClient internals couples the filesystem to implementation details
|
||||
- Bad, because token refresh coordination with TowerClient becomes manual
|
||||
|
||||
## Solution or decision outcome
|
||||
|
||||
Option 2 — NIO filesystem within nf-tower using TowerClient delegation. All HTTP calls go through `TowerClient.sendApiRequest()`, ensuring a single point of authentication and retry logic.
|
||||
|
||||
## Rationale & discussion
|
||||
|
||||
### Path Hierarchy
|
||||
|
||||
The `seqera://` path encodes the Platform's organizational structure directly:
|
||||
|
||||
```
|
||||
seqera:// → ROOT (directory, depth 0)
|
||||
└── <org>/ → ORGANIZATION (directory, depth 1)
|
||||
└── <workspace>/ → WORKSPACE (directory, depth 2)
|
||||
└── datasets/ → RESOURCE TYPE (directory, depth 3)
|
||||
└── <name>[@<version>] → DATASET (file, depth 4)
|
||||
```
|
||||
|
||||
Each level is a directory except the leaf dataset, which is a file. Version pinning uses an `@version` suffix on the dataset name segment (e.g. `seqera://acme/research/datasets/samples@2`). Without it, the latest non-disabled version is resolved.
|
||||
|
||||
### Name-to-ID Resolution
|
||||
|
||||
The path uses human-readable names but the Platform API requires numeric IDs. Resolution is built from two API calls at filesystem initialization:
|
||||
|
||||
1. `GET /user-info` → obtain `userId`
|
||||
2. `GET /user/{userId}/workspaces` → returns all accessible org/workspace pairs
|
||||
|
||||
This single source provides both directory listing content and name→ID mapping. Results are cached in `SeqeraFileSystem` with invalidation on write operations. `GET /orgs` is intentionally not used as it returns all platform orgs, not scoped to user membership.
|
||||
|
||||
### Component Structure
|
||||
|
||||
```
|
||||
plugins/nf-tower/src/main/io/seqera/tower/plugin/
|
||||
├── fs/ ← NIO layer
|
||||
│ ├── SeqeraFileSystemProvider ← FileSystemProvider (scheme: "seqera")
|
||||
│ ├── SeqeraFileSystem ← FileSystem with org/workspace/dataset caches
|
||||
│ ├── SeqeraPath ← Path implementation (depth 0–4)
|
||||
│ ├── SeqeraFileAttributes ← BasicFileAttributes
|
||||
│ ├── SeqeraPathFactory ← PF4J FileSystemPathFactory extension
|
||||
│ └── DatasetInputStream ← SeekableByteChannel over InputStream
|
||||
├── dataset/ ← API client layer
|
||||
│ ├── SeqeraDatasetClient ← Typed HTTP client wrapping TowerClient
|
||||
│ ├── DatasetDto ← Dataset API response model
|
||||
│ ├── DatasetVersionDto ← Version API response model
|
||||
│ ├── OrgAndWorkspaceDto ← Org/workspace list model
|
||||
│ └── WorkspaceOrgDto ← Workspace/org mapping model
|
||||
└── resources/META-INF/services/
|
||||
└── java.nio.file.spi.FileSystemProvider
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **TowerClient delegation**: `SeqeraDatasetClient` delegates all HTTP through `TowerFactory.client()` → `TowerClient.sendApiRequest()`. This ensures shared authentication state and avoids the token refresh corruption that would occur with separate HTTP client instances.
|
||||
|
||||
2. **One filesystem per JVM**: `SeqeraFileSystemProvider` maintains a single `SeqeraFileSystem` keyed by scheme. This matches the `TowerClient` singleton-per-session pattern.
|
||||
|
||||
3. **Read-only initial scope**: The filesystem reports `isReadOnly()=true`. Write support (dataset upload via multipart POST) is deferred to a future iteration.
|
||||
|
||||
4. **Download filename constraint**: The Platform API's download endpoint (`GET /datasets/{id}/v/{version}/n/{fileName}`) requires the exact filename from upload time. The implementation always resolves `DatasetVersionDto.fileName` from `GET /datasets/{id}/versions` before constructing the download URL.
|
||||
|
||||
5. **Extensible resource types**: The path hierarchy reserves depth 3 for a resource type segment (currently only `datasets`). Adding support for data-links or other resource types requires only a new handler at the directory listing and I/O layers, with no changes to path resolution or authentication.
|
||||
|
||||
6. **Thread safety**: `SeqeraFileSystem` cache methods and `SeqeraFileSystemProvider` lifecycle methods are `synchronized`. The filesystem map uses `LinkedHashMap` with external synchronization rather than `ConcurrentHashMap`, matching the low-contention access pattern.
|
||||
|
||||
### Limitations
|
||||
|
||||
- **No size metadata**: `SeqeraFileAttributes.size()` returns 0 for all paths because the Platform API does not expose content length in dataset metadata.
|
||||
- **Single endpoint per JVM**: The filesystem key is scheme-only; concurrent access to different Platform endpoints in the same JVM is not supported.
|
||||
|
||||
### Streaming Downloads
|
||||
|
||||
Dataset downloads use `TowerClient.sendStreamingRequest()` which calls `HxClient.sendAsStream()` — the response body is returned as an `InputStream` streamed directly from the HTTP connection. This avoids the triple-buffering problem (`String` → `getBytes()` → `ByteArrayInputStream`) that would otherwise consume ~40 MB heap per 10 MB dataset. The `HxClient.sendAsStream()` method goes through the same `sendWithRetry()` path as `sendAsString()`, so retry logic and token refresh are preserved.
|
||||
|
||||
## Links
|
||||
|
||||
- [Spec](../specs/260310-seqera-dataset-fs/spec.md)
|
||||
- [Implementation plan](../specs/260310-seqera-dataset-fs/plan.md)
|
||||
- [Data model](../specs/260310-seqera-dataset-fs/data-model.md)
|
||||
555
nextflow/adr/20260310-typed-workflows.md
Normal file
555
nextflow/adr/20260310-typed-workflows.md
Normal file
@@ -0,0 +1,555 @@
|
||||
# Typed workflows
|
||||
|
||||
- Authors: Ben Sherman
|
||||
- Status: accepted
|
||||
- Date: 2026-03-10
|
||||
- Tags: lang, static-types, workflows
|
||||
|
||||
## Summary
|
||||
|
||||
Extend workflows and dataflow logic to provide first-class support for static typing and records.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Workflow logic in Nextflow consists of composing processes, channels, and *dataflow operators* (or just *operators*). Operators are essential for transforming, filtering, and combining channels to control the flow of data through a pipeline.
|
||||
|
||||
However, workflows were not originally designed with static typing in mind. The introduction of static typing throughout the rest of the language has revealed several gaps in the design of workflow logic.
|
||||
|
||||
### Operators
|
||||
|
||||
Many operators cannot be statically type-checked because they do not have well-defined argument types and return types. Operators such as `combine` and `join` are designed to work with tuples, but do not support records.
|
||||
|
||||
A broader issue is that the operator library is very large, which makes it difficult to find the right operator for a given situation. Several operators deal with additional concerns such as reading/writing specific data formats, which blurs the distinction between dataflow logic and the domain-specific aspects of a workflow. Several operators rely on the ordering of values in a channel, which can cause non-deterministic behavior and hinder reproducibility.
|
||||
|
||||
The need for static typing is also an opportunity to address these issues by encouraging the use of a core subset of operators that provide all necessary functionality and support static typing.
|
||||
|
||||
### Dataflow syntax
|
||||
|
||||
There are many different ways to express the same dataflow logic. Consider the following example:
|
||||
|
||||
```groovy
|
||||
ch_input = channel.of('Hello', 'Hola', 'Ciao')
|
||||
|
||||
// alt 1
|
||||
ch_input
|
||||
| GREET
|
||||
| map { v -> v.toUpperCase() }
|
||||
| view
|
||||
| set { ch_upper }
|
||||
|
||||
// alt 2
|
||||
GREET(ch_input)
|
||||
GREET.out
|
||||
.map { v -> v.toUpperCase() }
|
||||
.tap { ch_upper }
|
||||
.view()
|
||||
|
||||
// alt 3
|
||||
ch_greet = GREET(ch_input)
|
||||
ch_upper = ch_greet
|
||||
.map { v -> v.toUpperCase() }
|
||||
.view()
|
||||
```
|
||||
|
||||
Here we see several syntax variants:
|
||||
|
||||
- Processes and operators can be composed with pipes (alt 1) or with method calls (alt 2, alt 3).
|
||||
|
||||
- Channels can be assigned using `set`, `tap`, or a regular assignment.
|
||||
|
||||
- Process outputs can be accessed using the `.out` property on the process name (alt 2) or by assignment (alt 3). The `.out` property can refer to a single output or a record of outputs, depending on the process definition.
|
||||
|
||||
Every syntax variant has a cost -- it make code look less familiar to new users, it can cause counterproductive debates over which variant is "better", and it makes Nextflow code less consistent overall. Even if you stick to your preferred syntax, you still have to learn the other variants because you might encounter them when reading someone else's code.
|
||||
|
||||
Therefore, syntax sugar should be used judiciously -- it should provide some value that makes adding it worth the aforementioned cost. The variants shown in alt 1 and alt 2 do not add much value relative to their cost.
|
||||
|
||||
Even the pipe (`|`), which is loved by many users, can rarely be used in its ideal form because processes usually have additional arguments that can’t be specified in a pipe chain.
|
||||
|
||||
## Goals
|
||||
|
||||
- Introduce first-class support for static typing and records with dataflow operators
|
||||
|
||||
- Encourage the use of a core set of operators (`map`, `filter`, `join`, etc)
|
||||
|
||||
- Discourage the use of non-deterministic operators (`buffer`, `distinct`, `first`, etc)
|
||||
|
||||
- Discourage the use of syntax variants that do not provide sufficient value to the language
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Remove support for existing workflow syntax and semantics -- static typing should be opt-in
|
||||
|
||||
- Change the way that processes are called -- processes are still called directly with channels, preserving the common mental model of "processes connected by channels"
|
||||
|
||||
## Solution
|
||||
|
||||
Introduce **typed workflows**, which provide a streamlined syntax for workflows that supports static typing.
|
||||
|
||||
Typed workflows can be used with the `nextflow.enable.types` feature flag:
|
||||
|
||||
```groovy
|
||||
// typed workflow
|
||||
nextflow.enable.types = true
|
||||
|
||||
workflow HELLO {
|
||||
take:
|
||||
ch_names: Channel<String>
|
||||
|
||||
main:
|
||||
ch_names.subscribe { name ->
|
||||
println "Hello, $name!"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```groovy
|
||||
// legacy workflow
|
||||
workflow HELLO {
|
||||
take:
|
||||
ch_names
|
||||
|
||||
main:
|
||||
ch_names.subscribe { name ->
|
||||
println "Hello, $name!"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This flag behaves the same way for typed processes and typed workflows:
|
||||
|
||||
- The flag must be specified in every script that uses typed processes/workflows
|
||||
- Typed processes/workflows cannot be mixed with legacy processes/workflows in the same script
|
||||
- Typed and non-typed scripts can be used in the same pipeline
|
||||
|
||||
### Operators
|
||||
|
||||
The operator library is extended to support static typing and records:
|
||||
|
||||
- The `combine` and `join` operators are extended to support both tuples and records.
|
||||
|
||||
- The `groupBy` operator is introduced as a statically-typed replacement for `groupTuple`
|
||||
|
||||
All operators can be used with or without static typing, with some caveats:
|
||||
|
||||
- Some operators have stricter semantics when static typing is enabled via `nextflow.enable.types`. These changes are necessary in order to support static typing effectively. They should not affect the majority of existing code.
|
||||
|
||||
- Some operators are discouraged from use with static typing. While they can still be used, the type checker will not be able to validate them. Users should be encouraged to migrate away from them in favor of the *core operators* that are statically typed.
|
||||
|
||||
The accompanying reference documentation and best practices guide explain these updates in detail. Here we highlight the most important changes.
|
||||
|
||||
The *core operators* are:
|
||||
|
||||
- `collect`: collect the channel values into a collection (dataflow value)
|
||||
- `combine`: emit the combinations of two channels
|
||||
- `filter`: emit only the channel values that satisfy a condition
|
||||
- `flatMap`: emit multiple values for each channel value with a closure
|
||||
- `groupBy`: group channel values by a grouping key
|
||||
- `join`: relational join of two channels based on a matching key
|
||||
- `map`: transform each channel value with a closure
|
||||
- `mix`: concatenate two channels
|
||||
- `reduce`: reduce channel values into a single value with an accumulator
|
||||
- `subscribe`: perform an action for each channel value
|
||||
- `unique`: emit unique values
|
||||
- `until`: emit each channel value until a stopping condition is satisfied
|
||||
- `view`: print each channel value
|
||||
|
||||
The core operators provide a minimal subset (13 out of ~50) that covers practically all use cases and supports static typing. Encouraging this subset as best practice makes it easier to find the right operator for a given situation, while preserving existing code that uses other operators.
|
||||
|
||||
The *legacy operators* are:
|
||||
|
||||
| Operator | Problem | Migration strategy |
|
||||
|---|---|---|
|
||||
| `branch` | Redundant | Use `filter` and `map` for each branch instead |
|
||||
| `buffer`, `collate` | Non-deterministic | Use `List::collate()` instead |
|
||||
| `collectFile` | Not statically typed | Use `collect`, `groupBy`, and `Iterable::toSorted()` instead |
|
||||
| `concat` | Redundant | Use `mix` instead |
|
||||
| `count`, `max`, `min`, `sum` | Redundant, rarely used | Use `collect` and the corresponding `Iterable` method instead |
|
||||
| `cross` | Redundant | Use `join` instead |
|
||||
| `distinct` | Non-deterministic | Use `unique` instead |
|
||||
| `dump` | Redundant | Use `view` with `tag` option instead |
|
||||
| `first`, `last`, `take` | Non-deterministic | Use a list instead |
|
||||
| `flatten` | Not statically typed | Use `flatMap` instead |
|
||||
| `groupTuple` | Not statically typed | Use `groupBy` instead |
|
||||
| `ifEmpty` | Not statically typed | Use `map` with `?:` instead |
|
||||
| `merge` | Non-deterministic | Use `join` instead |
|
||||
| `multiMap` | Redundant | Use `map` instead |
|
||||
| `randomSample` | Non-deterministic | - |
|
||||
| `set`, `tap` | Redundant | Use a regular assignment instead |
|
||||
| `splitCsv`, `splitFasta`, `splitFastq`, `splitJson`, `splitText` | Not statically typed | Use `flatMap` with the equivalent `Path` method instead |
|
||||
| `countCsv`, `countFasta`, `countFastq`, `countJson`, `countLines` | Not statically typed | Use `flatMap` with the equivalent `Path` method instead |
|
||||
| `toDouble`, `toFloat`, `toInteger`, `toLong` | Redundant, rarely used | Use `map` and the corresponding `String` method instead |
|
||||
| `toList` | Redundant | Use `collect` instead |
|
||||
| `toSortedList` | Redundant | Use `collect` and `Iterable::toSorted()` instead |
|
||||
| `transpose` | Not statically typed | Use `flatMap` instead |
|
||||
|
||||
In most cases, a legacy operator can be rewritten in terms of core operators and standard library functions. The accompanying best practices guide provides detailed examples for each operator. Since legacy operators can still be used in typed workflows, users can migrate away from legacy operators at their own pace.
|
||||
|
||||
### Fewer syntax variants
|
||||
|
||||
Typed workflows do not support the following syntax variants:
|
||||
|
||||
- Implicit `it` closure parameter → declare an explicit parameter instead
|
||||
- `it` can still be used as a variable name as long as it is explicitly declared
|
||||
|
||||
- Using `Channel` to access channel factories → use `channel` instead
|
||||
- `Channel` should be used only in type annotations
|
||||
|
||||
- Using `set` or `tap` to assign channels → use assignments instead
|
||||
|
||||
- Special dataflow operators `|` and `&` → use assignments and method calls instead
|
||||
- The equivalent bitwise operators are still allowed
|
||||
|
||||
- Using the `.out` property to access process and workflow outputs → use assignments instead
|
||||
|
||||
These restrictions are designed to make Nextflow code more consistent across the board and more familiar to users from other programming languages. Things like variable assignments and method calls in Nextflow look and feel the same as most other languages, whereas things like `set` assignments and the `.out` property make Nextflow code feel more unfamiliar without adding much value.
|
||||
|
||||
This aspect of the language is becoming more salient as code is increasingly read and written by AI agents. Agents need many examples of a programming language in order to use it effectively, so when a niche language has many syntax variants or syntax that deviates heavily from the common patterns used by other languages, it hurts the agent's ability to read and write code in that language.
|
||||
|
||||
## Distinguishing between typed and legacy workflows
|
||||
|
||||
Static typing has been introduced as multiple independent features:
|
||||
|
||||
- Typed parameters (`params` block)
|
||||
- Typed outputs (`output` block)
|
||||
- Typed processes
|
||||
- Record types
|
||||
- Typed workflows (this proposal)
|
||||
|
||||
This incremental approach was done in contrast to DSL2, which was a monolithic change that required an entire pipeline to be updated at once. With static typing, each new feature can be adopted independently of the others, rather than requiring all new features to be adopted at once (e.g. "DSL3").
|
||||
|
||||
Most of the features for static typing are new concepts that can be used alongside existing code. However, typed processes and typed workflows modify existing concepts (`process` and `workflow` definitions), so they require a feature flag.
|
||||
|
||||
The `nextflow.enable.types` feature flag will be used to distinguish between typed and legacy code, indefinitely. It would only be removed if the support for legacy syntax was removed, which is unlikely since DSL2 has been the standard Nextflow syntax for many years.
|
||||
|
||||
To help distinguish between typed and legacy workflows, the use of type annotations should be allowed only for typed workflows:
|
||||
|
||||
```groovy
|
||||
// legacy workflow
|
||||
workflow greet {
|
||||
take:
|
||||
greetings
|
||||
|
||||
main:
|
||||
messages = greetings.map { v -> "$v world!" }
|
||||
|
||||
emit:
|
||||
messages
|
||||
}
|
||||
```
|
||||
|
||||
```groovy
|
||||
// typed workflow
|
||||
nextflow.enable.types = true
|
||||
|
||||
workflow greet {
|
||||
take:
|
||||
greetings: Channel<String>
|
||||
|
||||
main:
|
||||
messages = greetings.map { v -> "$v world!" }
|
||||
|
||||
emit:
|
||||
messages: Channel<String>
|
||||
}
|
||||
```
|
||||
|
||||
## Interoperability between typed and legacy workflows
|
||||
|
||||
Typed and legacy workflows use different underlying dataflow types:
|
||||
|
||||
- **Legacy workflows (v1)** use raw GPars types: `DataflowBroadcast` (queue channel) and `DataflowVariable` (value channel).
|
||||
|
||||
- **Typed workflows (v2)** use wrapper types: `ChannelImpl` (wraps a `DataflowBroadcast`) and `ValueImpl` (wraps a `DataflowVariable`). These wrappers implement the new operators and integrate with the type system.
|
||||
|
||||
While a given script must be entirely typed or entirely legacy (controlled by the `nextflow.enable.types` flag), **typed and legacy workflows can call each other across different scripts**. This interoperability enables incremental migration -- individual scripts can be migrated to static typing without having to update the entire pipeline at once.
|
||||
|
||||
### Normalization at call sites
|
||||
|
||||
When a workflow calls another workflow, the Nextflow runtime automatically converts dataflow arguments and return values to the appropriate type for each side of the call site.
|
||||
|
||||
Normalization can occur in either direction:
|
||||
|
||||
- **v2 → v1 (unwrap)**: when passing typed channels to a legacy component, `ChannelImpl` / `ValueImpl` are unwrapped to the underlying `DataflowBroadcast` / `DataflowVariable`.
|
||||
|
||||
- **v1 → v2 (wrap)**: when passing legacy channels to a typed component, `DataflowBroadcast` / `DataflowVariable` are wrapped as `ChannelImpl` / `ValueImpl`.
|
||||
|
||||
The normalization is applied twice per call: once to the arguments (converted to match the *callee's* semantics), and once to the return value (converted to match the *caller's* semantics).
|
||||
|
||||
### Example: typed workflow calling a legacy workflow
|
||||
|
||||
**`legacy.nf`**
|
||||
```groovy
|
||||
workflow LEGACY_ALIGN {
|
||||
take:
|
||||
reads // DataflowBroadcast
|
||||
|
||||
main:
|
||||
ALIGN(reads)
|
||||
|
||||
emit:
|
||||
bam = ALIGN.out // DataflowBroadcast
|
||||
}
|
||||
```
|
||||
|
||||
**`typed.nf`**
|
||||
```groovy
|
||||
nextflow.enable.types = true
|
||||
|
||||
include { LEGACY_ALIGN } from './legacy'
|
||||
|
||||
workflow {
|
||||
reads = channel.fromPath('*.fastq') // ChannelImpl
|
||||
|
||||
// `reads` is unwrapped to DataflowBroadcast when passed to LEGACY_ALIGN
|
||||
// The return value (DataflowBroadcast) is wrapped to ChannelImpl
|
||||
bam = LEGACY_ALIGN(reads)
|
||||
|
||||
bam.view() // ChannelImpl
|
||||
}
|
||||
```
|
||||
|
||||
### Example: legacy workflow calling a typed workflow
|
||||
|
||||
**`typed.nf`**
|
||||
```groovy
|
||||
nextflow.enable.types = true
|
||||
|
||||
workflow TYPED_TRIM {
|
||||
take:
|
||||
reads: Channel<Record>
|
||||
|
||||
main:
|
||||
ch_trimmed = TRIM(reads)
|
||||
|
||||
emit:
|
||||
trimmed = ch_trimmed
|
||||
}
|
||||
```
|
||||
|
||||
**`legacy.nf`**
|
||||
```groovy
|
||||
include { TYPED_TRIM } from './typed'
|
||||
|
||||
workflow {
|
||||
reads = Channel.fromPath('*.fastq') // DataflowBroadcast
|
||||
|
||||
// `reads` is wrapped as ChannelImpl when passed to TYPED_TRIM
|
||||
// The return value (ChannelImpl) is unwrapped to DataflowBroadcast
|
||||
trimmed = TYPED_TRIM(reads)
|
||||
|
||||
trimmed.view() // DataflowBroadcast
|
||||
}
|
||||
```
|
||||
|
||||
### Process and workflow outputs (`ChannelOut`)
|
||||
|
||||
Processes and workflows -- regardless of whether they are legacy or typed -- always return a `ChannelOut`, a specialized class that can contain one or more named outputs (`DataflowBroadcast` / `DataflowVariable`).
|
||||
|
||||
When a `ChannelOut` is returned to a typed workflow, it is normalized as follows:
|
||||
|
||||
- If the `ChannelOut` contains only one output, it is unwrapped to the underlying `DataflowBroadcast` / `DataflowVariable` and then wrapped as a `ChannelImpl` / `ValueImpl`.
|
||||
|
||||
- If the `ChannelOut` contains multiple outputs, it is converted to a record (`RecordMap`), where each named output is wrapped as a `ChannelImpl` / `ValueImpl`.
|
||||
|
||||
For example:
|
||||
|
||||
**`legacy.nf`**
|
||||
```groovy
|
||||
workflow LEGACY_QC {
|
||||
take:
|
||||
reads
|
||||
|
||||
main:
|
||||
FASTQC(reads)
|
||||
MULTIQC(FASTQC.out)
|
||||
|
||||
emit:
|
||||
fastqc = FASTQC.out // DataflowBroadcast
|
||||
multiqc = MULTIQC.out // DataflowBroadcast
|
||||
}
|
||||
```
|
||||
|
||||
**`typed.nf`**
|
||||
```groovy
|
||||
nextflow.enable.types = true
|
||||
|
||||
include { LEGACY_QC } from './legacy'
|
||||
|
||||
workflow {
|
||||
reads = channel.fromPath('*.fastq')
|
||||
|
||||
// LEGACY_QC returns a ChannelOut with two outputs
|
||||
// which is converted to a record:
|
||||
// Record { fastqc: ChannelImpl ; multiqc: ChannelImpl }
|
||||
qc = LEGACY_QC(reads)
|
||||
|
||||
// RecordMap provides same semantics as ChannelOut
|
||||
qc.fastqc.view()
|
||||
qc.multiqc.view()
|
||||
}
|
||||
```
|
||||
|
||||
## Alternatives
|
||||
|
||||
### Processes in operator closures
|
||||
|
||||
A process call is essentially a task function wrapped in a `map` operation. But processes are called directly on channels, which has a few implications:
|
||||
|
||||
- The true structure of process calls are somewhat obscured
|
||||
- Process calls can have a different return type (`Channel` or `Value`) depending on how they are called
|
||||
- Processes can only be called as a `map` operation, not with other operators like `reduce`
|
||||
- Processes can not be chained like operator calls (without additional syntax like `|`)
|
||||
|
||||
These limitations could be addressed by calling processes in operator closures instead of calling them directly with channels:
|
||||
|
||||
```groovy
|
||||
ch_samples = channel.of(...)
|
||||
fasta = file(...)
|
||||
|
||||
// before
|
||||
SALMON(ch_samples, fasta)
|
||||
|
||||
// after
|
||||
ch_samples.map { sample -> SALMON(sample, fasta) }
|
||||
```
|
||||
|
||||
Where `SALMON` is defined as follows:
|
||||
|
||||
```groovy
|
||||
process SALMON {
|
||||
input:
|
||||
record(
|
||||
id: String,
|
||||
fastq: Path
|
||||
)
|
||||
fasta: Path
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
This syntax brings a number of benefits:
|
||||
|
||||
- The `map` operation is explicit, making process calls consistent with other operator logic
|
||||
- The process call matches the process definition -- it accepts and returns regular values, not channels or dataflow values
|
||||
- Processes can be chained without needing a pipe syntax (e.g. `ch.map(FOO).map(BAR).map(BAZ) ...`)
|
||||
- Processes could theoretically be composed with other operators (e.g. an iterative process with the `reduce` operator)
|
||||
- The closure around the process call can be used to handle process inputs and outputs without additional operator calls
|
||||
|
||||
Decoupling the process lifecycle from an implicit `map` operation, however, breaks a key assumption of the Nextflow runtime:
|
||||
|
||||
- Handling a process call in an arbitrary closure instead of as an implicit `map` operation is significantly more complex, and would likely require new language semantics and compiler transformations to implement.
|
||||
- Alternatively, the compiler could restrict such closures to specific patterns (e.g. a single process call with some statements before and after), but this would add complexity for developer experience (i.e. having to remember which patterns are allowed in which cases).
|
||||
|
||||
Additionally, some of the problems that motivated this approach have been addressed by type checking and records:
|
||||
|
||||
- The type checker can infer the return type of a direct process call from the call arguments (e.g. `Channel` vs `Value`)
|
||||
- Records and record types provide additional flexibility that eliminates much of the adaptor logic that was required between tuple channels and processes
|
||||
|
||||
Ultimately, this change would mostly be a cosmetic syntax improvement that would do little to improve the developer experience, but would introduce a great deal of complexity to the compiler and runtime. It would also be a significant break from the way that Nextflow workflows have been written since the introduction of DSL2.
|
||||
|
||||
### Processes as operator closures
|
||||
|
||||
A moderated version of calling processes in operator closures is to call them *as* operator closures:
|
||||
|
||||
```groovy
|
||||
ch_samples = channel.of(...)
|
||||
fasta = file(...)
|
||||
|
||||
// before
|
||||
SALMON(ch_samples, fasta)
|
||||
|
||||
// after
|
||||
ch_samples.map(SALMON, index: fasta)
|
||||
```
|
||||
|
||||
The process name takes the place of the `map` closure. The channel calling `map` is supplied as the first process input, and any additional inputs are supplied as named arguments to `map`.
|
||||
|
||||
This approach avoids much of the aforementioned complexity risk while retaining many of the benefits.
|
||||
|
||||
For example, process calls can be chained with other operator calls:
|
||||
|
||||
```groovy
|
||||
ch_input
|
||||
.map(GREET, greeting: "Hello")
|
||||
.map { v -> v.toUpperCase() }
|
||||
.view()
|
||||
```
|
||||
|
||||
And processes can be called with other operators such as `reduce`:
|
||||
|
||||
```groovy
|
||||
process ACCUMULATE {
|
||||
input:
|
||||
result: Path
|
||||
input: Path
|
||||
|
||||
script:
|
||||
"""
|
||||
cat ${input} >> ${result}
|
||||
"""
|
||||
|
||||
output:
|
||||
file('result.txt')
|
||||
}
|
||||
|
||||
workflow {
|
||||
channel.fromPath("*.txt").reduce(ACCUMULATE).view()
|
||||
}
|
||||
```
|
||||
|
||||
This particular pattern was proposed as a cleaner alternative to the experimental [recursion](https://nextflow.io/docs/latest/workflow.html#process-and-workflow-recursion) feature. As long as the process matches the signature of the accumulator closure (two inputs and one output), the process can be executed iteratively.
|
||||
|
||||
While this approach avoids most of the potential complexity that would be required to call processes in operator closures, it is still a significant syntax change with dubious relative benefit.
|
||||
|
||||
Investigating these approaches revealed an important trade-off -- Nextflow sacrifices a small amount of syntactic precision in order to make process calls prominent in the workflow logic. While calling processes in an operator would be more correct and provide some additional flexibility (e.g. using processes with other operators), it would make workflows feel much more like "operators that call processes in closures" instead of "processes connected by channels".
|
||||
|
||||
The reality is that most Nextflow users think of their pipelines as "processes connected by channels", and operator logic is a minor detail at best and a confusing distraction at worst. While we can and should make channel operators as simple and pleasant to use as possible, it should be in service of making them less prominent in the language, not more.
|
||||
|
||||
### Implicit dataflow values
|
||||
|
||||
Dataflow values (a.k.a. *value channels*) are analogous to Futures or Promises in other languages. For example, given a `CompletableFuture` in Java, you can either call `get()` to await the value or `thenAccept()` / `thenApply()` to invoke a callback when the value is ready.
|
||||
|
||||
Dataflow values can similarly call `subscribe` or `map`, but it is not possible to "await" a dataflow value directly. For example, it is not possible to use a dataflow value in an `if` statement:
|
||||
|
||||
```groovy
|
||||
vals = channel.of(1..10).collect()
|
||||
if( vals.size() > 2 )
|
||||
println 'More than two!'
|
||||
```
|
||||
|
||||
Instead, you must use `subscribe` to act on the value asynchronously:
|
||||
|
||||
```groovy
|
||||
vals = channel.of(1..10).collect()
|
||||
vals.subscribe { _vals ->
|
||||
if( _vals.size() > 2 )
|
||||
println 'More than two!'
|
||||
}
|
||||
```
|
||||
|
||||
This is a common frustration for many users, that dataflow values don't quite work like regular values, even though it seems like they should.
|
||||
|
||||
A solution could be to make dataflow values *implicit* -- users would use them like regular values (i.e. the first example above) and the compiler would translate the user's code into explicit dataflow logic (i.e. the second example).
|
||||
|
||||
To do this, the compiler would need to:
|
||||
|
||||
1. distinguish implicit dataflow values from regular values via type inference (e.g. the result of a `collect` operator),
|
||||
|
||||
2. wrap downstream code in `map` and/or `subscribe` operators as needed to produce the desired dataflow logic.
|
||||
|
||||
In the end, however, this change does not seem worthwhile:
|
||||
|
||||
- It makes type inference an essential part of the compilation process rather than an optional enhancement.
|
||||
|
||||
- The above example is simple to understand, but it is easy to construct more complicated examples that quickly cast doubt on whether the compiler could solve this problem in general.
|
||||
|
||||
- Even if there is a general solution, any mistake or edge case would likely lead to unexpected behavior that would be extremely difficult to debug (e.g. a low-level compiler error, compiled code that is silently incorrect).
|
||||
|
||||
Additionally, most of the problems that motivated this idea have been effectively solved by type checking:
|
||||
|
||||
- There is now an explicit `Value` type which allows both developers and the type checker to distinguish between channels and dataflow values.
|
||||
|
||||
- While users still can't use a dataflow value in an `if` statement, they can get clear and early feedback on whether their code is valid, which is what ultimately matters.
|
||||
|
||||
- Being transparent about regular values vs dataflow values in the language may be for the best anyway -- it provides a clear picture of how things are working "under the hood", and it is still far simpler than the async programming models employed by most languages.
|
||||
181
nextflow/adr/20260322-platform-oidc-pkce-login.md
Normal file
181
nextflow/adr/20260322-platform-oidc-pkce-login.md
Normal file
@@ -0,0 +1,181 @@
|
||||
# Replace Auth0 with Platform OIDC PKCE for Nextflow CLI login
|
||||
|
||||
- Authors: Paolo Di Tommaso
|
||||
- Status: draft
|
||||
- Date: 2026-03-22
|
||||
- Tags: auth, oidc, pkce, nextflow
|
||||
|
||||
## Summary
|
||||
|
||||
Replace the Auth0 Device Authorization Grant used by `nextflow auth login` with an OAuth2 Authorization Code + PKCE flow against Seqera Platform itself acting as the OIDC identity provider. Implement the reusable OIDC PKCE client as `lib-platform-oidc` in libseqera.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The `nextflow auth login` command authenticates via Auth0, an external identity provider, using the OAuth2 Device Authorization Grant. This requires hardcoded Auth0 domain and client ID mappings per environment (dev, stage, prod) and creates an artificial distinction between "cloud" and "enterprise" endpoints — cloud uses Auth0 while enterprise falls back to manual PAT entry.
|
||||
|
||||
Seqera Platform now exposes a standards-compliant OIDC provider at `/.well-known/openid-configuration`, making it possible for CLI clients to authenticate directly against Platform using Authorization Code + PKCE — eliminating the Auth0 dependency and unifying the login flow for all Platform instances.
|
||||
|
||||
## Goals
|
||||
|
||||
- Single login flow for all Platform endpoints (cloud and enterprise)
|
||||
- Eliminate Auth0 dependency and hardcoded domain/clientId mappings
|
||||
- Deprecate `TOWER_AUTH_DOMAIN` and `TOWER_AUTH_CLIENT_ID` env vars
|
||||
- Reusable OIDC PKCE library in libseqera for other CLI tools
|
||||
- No changes to PAT generation, storage, runtime usage, refresh, or logout
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Changing how PATs are generated or stored after login
|
||||
- Modifying runtime token refresh (`TowerXAuth`, `TowerClient`, `WaveClient`)
|
||||
- Removing backward compatibility with existing PATs in config
|
||||
|
||||
Note: The logout flow was simplified as a consequence of removing the cloud-vs-enterprise distinction — PAT deletion via Platform API is now always attempted for all endpoints.
|
||||
|
||||
## Login Flow
|
||||
|
||||
```
|
||||
Nextflow CLI Browser Platform
|
||||
│ │
|
||||
│ GET /.well-known/openid-configuration │
|
||||
│───────────────────────────────────────────────────────────────────>│
|
||||
│<───────────── { authorization_endpoint, token_endpoint } ──────────│
|
||||
│ │
|
||||
│ [generate code_verifier, code_challenge, state] │
|
||||
│ [start local HTTP server on 127.0.0.1:PORT] │
|
||||
│ │
|
||||
│ open browser ──>│ │
|
||||
│ │ GET /authorize? │
|
||||
│ │ client_id=nextflow_cli │
|
||||
│ │ &response_type=code │
|
||||
│ │ &scope=openid+profile+email+offline_access │
|
||||
│ │ &redirect_uri=http://127.0.0.1:PORT/callback │
|
||||
│ │ &state=<random> │
|
||||
│ │ &code_challenge=<S256(verifier)> │
|
||||
│ │ &code_challenge_method=S256 │
|
||||
│ │────────────────────────────────────────────────>│
|
||||
│ │ │
|
||||
│ │ (user authenticates on Platform) │
|
||||
│ │ │
|
||||
│ │<── redirect to 127.0.0.1:PORT/callback ─────────│
|
||||
│ │ ?code=<auth_code>&state=<state> │
|
||||
│ │ │
|
||||
│ [callback server receives code, validates state] │
|
||||
│ [returns HTML "Login successful" to browser] │
|
||||
│ │
|
||||
│ POST /token │
|
||||
│ grant_type=authorization_code │
|
||||
│ &client_id=nextflow_cli │
|
||||
│ &code=<auth_code> │
|
||||
│ &code_verifier=<verifier> │
|
||||
│ &redirect_uri=http://127.0.0.1:PORT/callback │
|
||||
│───────────────────────────────────────────────────────────────────>│
|
||||
│<──────────── { access_token, refresh_token } ──────────────────────│
|
||||
│ │
|
||||
│ GET /user-info (Authorization: Bearer <access_token>) │
|
||||
│───────────────────────────────────────────────────────────────────>│
|
||||
│<──────────────────── { user info } ────────────────────────────────│
|
||||
│ │
|
||||
│ POST /tokens (generate PAT — same as current flow) │
|
||||
│───────────────────────────────────────────────────────────────────>│
|
||||
│<──────────────────── { accessKey: <PAT> } ─────────────────────────│
|
||||
│ │
|
||||
│ [save PAT to ~/.nextflow/seqera-auth.config] │
|
||||
```
|
||||
|
||||
## Solution
|
||||
|
||||
### 1. Register `nextflow_cli` client in Platform
|
||||
|
||||
**File:** `platform/tower-config/src/main/resources/application-oauth-client.yml`
|
||||
|
||||
```yaml
|
||||
- client-id: "nextflow_cli"
|
||||
client-name: "Nextflow CLI"
|
||||
client-type: NATIVE
|
||||
client-secret: null
|
||||
application-type: "native"
|
||||
token-endpoint-auth-method: "none"
|
||||
id-token-signed-response-alg: "RS256"
|
||||
redirect-uris:
|
||||
- "http://127.0.0.1"
|
||||
allowed-flows:
|
||||
- "authorization_code_pkce"
|
||||
allowed-scopes:
|
||||
- "openid"
|
||||
- "profile"
|
||||
- "email"
|
||||
- "offline_access"
|
||||
third-party: false
|
||||
```
|
||||
|
||||
### 2. New module `lib-platform-oidc` in libseqera
|
||||
|
||||
Plain Java production code, Groovy/Spock tests. No external dependencies — uses only JDK classes (`java.net.http.HttpClient`, `com.sun.net.httpserver.HttpServer`, `java.security.*`).
|
||||
|
||||
```
|
||||
lib-platform-oidc/
|
||||
build.gradle
|
||||
VERSION
|
||||
changelog.txt
|
||||
src/
|
||||
main/java/io/seqera/platform/auth/oidc/
|
||||
OidcConfig.java # authorization_endpoint + token_endpoint
|
||||
PkceChallenge.java # code_verifier + code_challenge
|
||||
PkceUtil.java # PKCE generation helpers
|
||||
OidcDiscovery.java # GET /.well-known/openid-configuration
|
||||
OidcCallbackServer.java # Local HTTP server on 127.0.0.1:0
|
||||
OidcTokenExchange.java # POST token endpoint for code exchange
|
||||
OidcLoginFlow.java # Orchestrator
|
||||
test/groovy/io/seqera/platform/auth/oidc/
|
||||
PkceUtilTest.groovy
|
||||
OidcDiscoveryTest.groovy
|
||||
OidcCallbackServerTest.groovy
|
||||
OidcTokenExchangeTest.groovy
|
||||
OidcLoginFlowTest.groovy
|
||||
```
|
||||
|
||||
**`OidcLoginFlow`** — public API:
|
||||
```java
|
||||
public class OidcLoginFlow {
|
||||
public OidcLoginFlow(String endpoint, String clientId) { ... }
|
||||
public String login(Consumer<String> browserLauncher) throws Exception { ... }
|
||||
}
|
||||
```
|
||||
|
||||
Flow: OIDC discovery → generate PKCE → start callback server (ephemeral port) → invoke `browserLauncher` with authorization URL → wait for callback → exchange code for tokens → return `access_token`.
|
||||
|
||||
The `browserLauncher` callback delegates browser-opening to the caller since it's platform-specific.
|
||||
|
||||
### 3. Modify `AuthCommandImpl.login()` in Nextflow
|
||||
|
||||
**File:** `nextflow/plugins/nf-tower/src/main/io/seqera/tower/plugin/auth/AuthCommandImpl.groovy`
|
||||
|
||||
Replace the cloud-vs-enterprise branching with a single OIDC flow:
|
||||
|
||||
```groovy
|
||||
// Was: getCloudEndpointInfo → performAuth0Login / handleEnterpriseAuth
|
||||
// Now:
|
||||
performOidcLogin(apiUrl)
|
||||
```
|
||||
|
||||
`performOidcLogin()` creates an `OidcLoginFlow`, gets the OAuth access token, then follows the existing post-auth steps unchanged: `getUserInfo()` → `generatePAT()` → `saveAuthToConfig()` → `config()`.
|
||||
|
||||
Remove: `performAuth0Login()`, `requestDeviceAuthorization()`, `pollForDeviceToken()`, `performAuth0Request()`, `handleEnterpriseAuth()`, `promptPAT()`, `getCloudEndpointInfo()`.
|
||||
|
||||
### 4. Deprecate Auth0 mappings in PlatformHelper
|
||||
|
||||
**File:** `nextflow/modules/nextflow/src/main/groovy/nextflow/platform/PlatformHelper.groovy`
|
||||
|
||||
Add `@Deprecated` to `getAuthDomain()` and `getAuthClientId()`.
|
||||
|
||||
## Verification
|
||||
|
||||
1. `./gradlew :lib-platform-oidc:test` in libseqera
|
||||
2. `./gradlew :plugins:nf-tower:test` in Nextflow
|
||||
3. Manual: `./launch.sh auth login -url https://api.cloud.dev-seqera.io` — browser opens, PKCE flow completes, PAT stored in config
|
||||
4. Backward compat: existing PAT in `seqera-auth.config` continues to work
|
||||
|
||||
## Links
|
||||
|
||||
- [Platform OIDC provider PR](https://github.com/seqeralabs/platform/pull/10336)
|
||||
- [OAuth client examples PR](https://github.com/seqeralabs/platform/pull/10473)
|
||||
180
nextflow/adr/20260323-hints-process-directive.md
Normal file
180
nextflow/adr/20260323-hints-process-directive.md
Normal file
@@ -0,0 +1,180 @@
|
||||
# `hints` process directive for executor-specific scheduling hints
|
||||
|
||||
- Authors: Rob Syme
|
||||
- Status: accepted
|
||||
- Deciders: Paolo Di Tommaso, Ben Sherman, Rob Syme
|
||||
- Date: 2026-03-23
|
||||
- Tags: directive, executor, scheduling
|
||||
|
||||
## Summary
|
||||
|
||||
Introduce a `hints` process directive for executor-specific scheduling hints that don't map to existing directives.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Many executors can be configured in various ways on a per-task basis. For example:
|
||||
|
||||
- AWS Batch jobs can use *consumable resources* to limit concurrent job execution based on non-standard resources such as software license seats.
|
||||
|
||||
- Google Batch jobs can specify a *provisioning model* to control the use of spot vs on-demand VMs on a per-task basis.
|
||||
|
||||
- Seqera Scheduler supports a variety of resource and scheduling settings, including spot/on-demand provisioning.
|
||||
|
||||
These settings can be exposed by Nextflow as executor-specific config options, such as `google.batch.spot`, but config options are applied globally. In order to apply a setting to specific processes or tasks, it must be exposed as a process directive.
|
||||
|
||||
Process directives in Nextflow aim to provide a common vocabulary for executing tasks in many different environments. Directives such as `cpus`, `memory`, and `time` have broadly the same meaning across most executors, making it easier for users to write portable pipelines.
|
||||
|
||||
At the same time, many executors have custom settings not shared by other executors, and it is not practical to create a new process directive for every new setting. There are over 40 [process directives](https://docs.seqera.io/nextflow/reference/process#directives) at the time of writing, and every new directive adds cognitive load when a user is trying to find the right directive for a given situation.
|
||||
|
||||
There exist a few generic process directives already:
|
||||
|
||||
- The `clusterOptions` directive can be used to specify command-line arguments, primarily for HPC schedulers
|
||||
- The `ext` directive supports arbitrary key-values, but is designed primarily to customize the task script (e.g. tool arguments), not executor behavior
|
||||
- The `resourceLabels` directive also supports arbitrary key-values, but is intended for tagging and tracking resources, not controlling them
|
||||
|
||||
A new directive is needed to support executor-specific settings at a per-task level in a structured manner, without bloating the process directives for every new custom setting.
|
||||
|
||||
## Goals
|
||||
|
||||
- Provide a way to apply executor-specific settings to individual processes or tasks
|
||||
|
||||
- Avoid the proliferation of narrow, executor-specific directives (e.g. `consumableResources`, `schedulingPolicy`, etc.)
|
||||
|
||||
- Provide a single extension point that executors can consume selectively
|
||||
|
||||
- Allow settings to be specified as key-values, providing validation where possible
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Replacing existing directives (`cpus`, `memory`, `accelerator`, `queue`) — those remain the right place for standard resources
|
||||
|
||||
## Decision
|
||||
|
||||
Introduce a `hints` process directive with namespaced keys. Executors consume the hints they understand and silently ignore the rest.
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
### Syntax
|
||||
|
||||
The `hints` directive accepts a map of key-value pairs:
|
||||
|
||||
```groovy
|
||||
// process definition
|
||||
process runDragen {
|
||||
cpus 4
|
||||
memory '16 GB'
|
||||
hints consumableResources: ['my-dragen-license': 1, 'other-license': 2]
|
||||
|
||||
script:
|
||||
"""
|
||||
dragen --ref-dir /ref ...
|
||||
"""
|
||||
}
|
||||
```
|
||||
|
||||
```groovy
|
||||
// process config
|
||||
process {
|
||||
withName: 'runDragen' {
|
||||
hints = [
|
||||
consumableResources: ['my-dragen-license': 1, 'other-license': 2]
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Keys are strings. Values may be any raw data type: strings, numbers, booleans, lists, or maps. Executors are responsible for defining which hints they recognize and what value type each hint expects.
|
||||
|
||||
In the above example, the `consumableResources` hint is given as a map of resource name to quantity. The AWS Batch executor supplies it to each job request using `ConsumableResourceProperties`.
|
||||
|
||||
### Namespacing
|
||||
|
||||
Keys can use dot-separated scopes to namespace settings as needed:
|
||||
|
||||
```groovy
|
||||
hints consumableResources: ['my-dragen-license': 1]
|
||||
hints 'scheduling.priority': 10
|
||||
hints 'scheduling.provisioningModel': 'spot'
|
||||
```
|
||||
|
||||
Keys can be routed to specific executors by prefixing with the executor name and a slash (`/`):
|
||||
|
||||
```groovy
|
||||
hints 'awsbatch/consumableResources': ['my-dragen-license': 1]
|
||||
hints 'seqera/scheduling.provisioningModel': 'spot'
|
||||
hints 'k8s/nodeSelector': 'gpu=true'
|
||||
```
|
||||
|
||||
The executor prefix gives pipeline developers the ability to target specific executors and have assurance that it won't accidentally apply to other executors (e.g. if another executor adds support for the same hint in the future).
|
||||
|
||||
### Validation
|
||||
|
||||
Nextflow should validate hints to the best of its ability, to catch errors such as typos:
|
||||
|
||||
- **Prefixed hints** can be validated against the set of hints declared by the corresponding executor. Unrecognized hints should be reported as errors.
|
||||
|
||||
- **Unprefixed hints** can be validated against the union of hints declared by all executors. Since unprefixed hints might be supported by executors that aren't currently loaded, unrecognized hints should be reported as warnings.
|
||||
|
||||
### Multiple hint resolution
|
||||
|
||||
The `hints` directive uses *replacement semantics* when specified multiple times, meaning that each `hints` setting completely replaces any previous settings:
|
||||
|
||||
```groovy
|
||||
process {
|
||||
// generic hint
|
||||
hints = [provisioningModel: 'spot']
|
||||
|
||||
// specific hint replaces generic hint
|
||||
withLabel: 'dragen' {
|
||||
hints = [consumableResources: ['my-dragen-license': 1]]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Within a process definition, the `hints` directive uses *accumulation semantics*, meaning that subsequent `hints` directives are accumulated:
|
||||
|
||||
```groovy
|
||||
process runDragen {
|
||||
// multiple separate hints
|
||||
hints provisioningModel: 'spot'
|
||||
hints consumableResources: ['my-dragen-license': 1, 'other-license': 2]
|
||||
|
||||
// equivalent to...
|
||||
hints (
|
||||
provisioningModel: 'spot',
|
||||
consumableResources: ['my-dragen-license': 1, 'other-license': 2]
|
||||
)
|
||||
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
This behavior is consistent with other directives such as `pod` and `resourceLabels`. In practice, this means that a given `hints` setting should specify all relevant hints for the given context.
|
||||
|
||||
For example, the `withLabel` selector above should also specify the `provisioningModel` hint if the intention is to preserve that hint for the selected processes:
|
||||
|
||||
```groovy
|
||||
process {
|
||||
hints = [provisioningModel: 'spot']
|
||||
|
||||
withLabel: 'dragen' {
|
||||
hints = [provisioningModel: 'spot', consumableResources: ['my-dragen-license': 1]]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
While this approach may lead to duplication, it gives users and developers more control over which hints are applied in a given context.
|
||||
|
||||
### Initial hint catalog
|
||||
|
||||
The following hints should be supported initially:
|
||||
|
||||
| Hint name | Value type | Executors | Use case |
|
||||
|--|--|--|--|
|
||||
| `consumableResources` | `Map<String, Integer>` | AWS Batch | License-aware scheduling ([#5917](https://github.com/nextflow-io/nextflow/issues/5917)) |
|
||||
| `scheduling.priority` | `Integer` | AWS Batch | Job scheduling priority ([#6998](https://github.com/nextflow-io/nextflow/issues/6998)) |
|
||||
| `scheduling.provisioningModel` | `String` | Google Batch | Spot VM scheduling ([#3530](https://github.com/nextflow-io/nextflow/issues/3530)) |
|
||||
|
||||
## Links
|
||||
|
||||
- [Community issue](https://github.com/nextflow-io/nextflow/issues/5917)
|
||||
76
nextflow/adr/YYYYMMDD-template-name.md
Normal file
76
nextflow/adr/YYYYMMDD-template-name.md
Normal file
@@ -0,0 +1,76 @@
|
||||
# [short title of solved problem and solution]
|
||||
|
||||
- Authors: [who wrote the ADR]
|
||||
- Status: [draft | proposed | rejected | accepted | deprecated | … | superseded by [xxx](xxx.md)]
|
||||
- Deciders: [list everyone involved in the decision] <!-- optional - to be formalised -->
|
||||
- Date: [YYYY-MM-DD when the decision was last updated]
|
||||
- Tags: [space and/or comma separated list of tags]
|
||||
|
||||
Technical Story: [description | ticket/issue URL] <!-- optional -->
|
||||
|
||||
## Summary
|
||||
|
||||
Quick description of the problem and the context. Should not take more than 2-3 lines.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Description of the technical problem to solve or to decision to make. This should be concise but provide all required details and the context related to the technical decision to be taken.
|
||||
|
||||
## Goals or Decision Drivers
|
||||
|
||||
Depending the context define clearly what are the goals or what are the most important decision drivers.
|
||||
|
||||
- [driver 1, e.g., a force, facing concern, …]
|
||||
- [driver 2, e.g., a force, facing concern, …]
|
||||
- … <!-- numbers of drivers can vary -->
|
||||
|
||||
## Non-goals
|
||||
|
||||
Define what's out of the scope of this ADR.
|
||||
|
||||
## Considered Options <!-- optional -->
|
||||
|
||||
- [option 1]
|
||||
- [option 2]
|
||||
- [option 3]
|
||||
- … <!-- numbers of options can vary -->
|
||||
|
||||
|
||||
## Pros and Cons of the Options <!-- optional -->
|
||||
|
||||
### [option 1]
|
||||
|
||||
[example | description | pointer to more information | …] <!-- optional -->
|
||||
|
||||
- Good, because [argument a]
|
||||
- Good, because [argument b]
|
||||
- Bad, because [argument c]
|
||||
- … <!-- numbers of pros and cons can vary -->
|
||||
|
||||
### [option 2]
|
||||
|
||||
[example | description | pointer to more information | …] <!-- optional -->
|
||||
|
||||
- Good, because [argument a]
|
||||
- Good, because [argument b]
|
||||
- Bad, because [argument c]
|
||||
- … <!-- numbers of pros and cons can vary -->
|
||||
|
||||
|
||||
## Solution or decision outcome
|
||||
|
||||
Summarize the solution or decision outcome in one-two lines.
|
||||
|
||||
## Rationale & discussion
|
||||
|
||||
Describe the solution or the decision outcome discussing how decision drivers have been applied and how it matches the declared goals. This section is expected to be concise though providing comprehensive description of the technical solution and covering all uncertainty or ambiguous points.
|
||||
|
||||
## Links <!-- optional -->
|
||||
|
||||
- [Link type](link to adr) <!-- example: Refined by [xxx](yyyymmdd-xxx.md) -->
|
||||
- … <!-- numbers of links can vary -->
|
||||
|
||||
## More information
|
||||
|
||||
- [What is an ADR and why should you use them](https://github.com/thomvaill/log4brains/tree/master#-what-is-an-adr-and-why-should-you-use-them)
|
||||
- [ADR GitHub organization](https://adr.github.io/)
|
||||
416
nextflow/adr/module-spec-schema.json
Normal file
416
nextflow/adr/module-spec-schema.json
Normal file
@@ -0,0 +1,416 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://raw.githubusercontent.com/nextflow-io/schemas/main/module/v1/schema.json",
|
||||
"title": "Nextflow Module Schema",
|
||||
"description": "Schema for Nextflow module meta.yml files, supporting both nf-core community patterns and the Nextflow module system",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Module name. Can be a simple identifier (e.g., 'fastqc', 'bwa_mem') for local/nf-core modules, or a fully qualified scoped name (e.g., 'nf-core/fastqc', 'myorg/custom') for registry modules.",
|
||||
"examples": ["fastqc", "bwa_mem", "nf-core/fastqc", "myorg/salmon-quant"],
|
||||
"pattern": "^([a-z0-9][a-z0-9-]*/)?[a-z][a-z0-9_-]*$"
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"description": "Semantic version of the module (MAJOR.MINOR.PATCH). Required for registry publication",
|
||||
"pattern": "^(0|[1-9]\\d*)\\.(0|[1-9]\\d*)\\.(0|[1-9]\\d*)(-[0-9A-Za-z-]+(\\.[0-9A-Za-z-]+)*)?(\\+[0-9A-Za-z-]+(\\.[0-9A-Za-z-]+)*)?$",
|
||||
"examples": ["1.0.0", "2.1.3", "1.0.0-beta.1"]
|
||||
},
|
||||
"description": {
|
||||
"type": "string",
|
||||
"description": "Brief description of what the module does",
|
||||
"minLength": 10,
|
||||
"maxLength": 500
|
||||
},
|
||||
"keywords": {
|
||||
"type": "array",
|
||||
"description": "Keywords for discovery and categorization",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"minLength": 2
|
||||
},
|
||||
"minItems": 1,
|
||||
"uniqueItems": true
|
||||
},
|
||||
"license": {
|
||||
"type": "string",
|
||||
"description": "SPDX license identifier for the module code itself",
|
||||
"examples": ["MIT", "Apache-2.0", "GPL-3.0-or-later"]
|
||||
},
|
||||
"authors": {
|
||||
"type": "array",
|
||||
"description": "Original authors of the module (GitHub handles preferred)",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"pattern": "^@?[a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?$"
|
||||
},
|
||||
"minItems": 1
|
||||
},
|
||||
"maintainers": {
|
||||
"type": "array",
|
||||
"description": "Current maintainers of the module (GitHub handles preferred)",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"pattern": "^@?[a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?$"
|
||||
}
|
||||
},
|
||||
"requires": {
|
||||
"type": "object",
|
||||
"description": "Runtime requirements for the module",
|
||||
"properties": {
|
||||
"nextflow": {
|
||||
"type": "string",
|
||||
"description": "Nextflow version constraint using comparison operators",
|
||||
"examples": [">=24.04.0", ">=24.04.0,<25.0.0"],
|
||||
"pattern": "^[<>=!]+[0-9]+\\.[0-9]+\\.[0-9]+(-[a-zA-Z0-9]+)?(,\\s*[<>=!]+[0-9]+\\.[0-9]+\\.[0-9]+(-[a-zA-Z0-9]+)?)*$"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
},
|
||||
"input": {
|
||||
"type": "array",
|
||||
"description": "Inputs of the module",
|
||||
"items": {
|
||||
"$ref": "#/$defs/structuredParameter"
|
||||
}
|
||||
},
|
||||
"output": {
|
||||
"type": "array",
|
||||
"description": "Outputs of the module",
|
||||
"items": {
|
||||
"$ref": "#/$defs/structuredParameter"
|
||||
}
|
||||
},
|
||||
"topics": {
|
||||
"type": "array",
|
||||
"description": "Topics of the module",
|
||||
"items": {
|
||||
"$ref": "#/$defs/structuredParameter"
|
||||
}
|
||||
},
|
||||
"tools": {
|
||||
"type": "array",
|
||||
"description": "Software tools wrapped by this module with their metadata",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"minProperties": 1,
|
||||
"maxProperties": 1,
|
||||
"patternProperties": {
|
||||
"^[a-zA-Z][a-zA-Z0-9_-]*$": {
|
||||
"$ref": "#/$defs/toolSpec"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": ["name", "description"],
|
||||
"$defs": {
|
||||
"toolSpec": {
|
||||
"type": "object",
|
||||
"description": "Specification for a software tool used by the module",
|
||||
"properties": {
|
||||
"description": {
|
||||
"type": "string",
|
||||
"description": "Description of the tool and its purpose"
|
||||
},
|
||||
"homepage": {
|
||||
"type": "string",
|
||||
"format": "uri",
|
||||
"description": "Tool's homepage URL",
|
||||
"pattern": "^https?://.*$"
|
||||
},
|
||||
"documentation": {
|
||||
"type": "string",
|
||||
"format": "uri",
|
||||
"description": "Documentation URL",
|
||||
"pattern": "^(https?|ftp)://.*$"
|
||||
},
|
||||
"tool_dev_url": {
|
||||
"type": "string",
|
||||
"format": "uri",
|
||||
"description": "Development/source code URL",
|
||||
"pattern": "^https?://.*$"
|
||||
},
|
||||
"doi": {
|
||||
"description": "Digital Object Identifier for the tool's publication",
|
||||
"oneOf": [
|
||||
{
|
||||
"type": "string",
|
||||
"pattern": "^10\\.\\d{4,9}/[^,]+$"
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"const": "no DOI available"
|
||||
}
|
||||
]
|
||||
},
|
||||
"arxiv": {
|
||||
"type": "string",
|
||||
"description": "arXiv identifier",
|
||||
"pattern": "^arXiv:\\d{4}\\.\\d{4,5}(v\\d+)?$"
|
||||
},
|
||||
"licence": {
|
||||
"type": "array",
|
||||
"description": "SPDX license identifier(s) for the tool",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"minItems": 1,
|
||||
"uniqueItems": true
|
||||
},
|
||||
"identifier": {
|
||||
"description": "bio.tools identifier or empty string",
|
||||
"oneOf": [
|
||||
{
|
||||
"type": "string",
|
||||
"pattern": "^biotools:[a-zA-Z0-9_-]+$"
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"maxLength": 0
|
||||
}
|
||||
]
|
||||
},
|
||||
"manual": {
|
||||
"type": "string",
|
||||
"format": "uri",
|
||||
"description": "Manual/user guide URL"
|
||||
}
|
||||
},
|
||||
"required": ["description"],
|
||||
"anyOf": [
|
||||
{
|
||||
"required": ["homepage"]
|
||||
},
|
||||
{
|
||||
"required": ["documentation"]
|
||||
},
|
||||
{
|
||||
"required": ["tool_dev_url"]
|
||||
},
|
||||
{
|
||||
"required": ["doi"]
|
||||
}
|
||||
]
|
||||
},
|
||||
"structuredParameter": {
|
||||
"oneOf": [
|
||||
{
|
||||
"$ref": "#/$defs/paramSpec"
|
||||
},
|
||||
{
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/paramSpec"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"paramSpec": {
|
||||
"type": "object",
|
||||
"description": "Specification for a module parameter",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Parameter identifier",
|
||||
"pattern": "^[a-zA-Z_][a-zA-Z0-9_]*$"
|
||||
},
|
||||
"type": {
|
||||
"type": "string",
|
||||
"description": "Data type of the parameter value",
|
||||
"enum": [
|
||||
"boolean",
|
||||
"float",
|
||||
"integer",
|
||||
"string",
|
||||
"list",
|
||||
"map",
|
||||
"file",
|
||||
"directory"
|
||||
]
|
||||
},
|
||||
"description": {
|
||||
"type": "string",
|
||||
"description": "Human-readable description of the parameter"
|
||||
},
|
||||
"pattern": {
|
||||
"type": "string",
|
||||
"description": "Glob pattern for file/directory parameters"
|
||||
},
|
||||
"optional": {
|
||||
"type": "boolean",
|
||||
"description": "Whether this parameter is optional",
|
||||
"default": false
|
||||
},
|
||||
"enum": {
|
||||
"type": "array",
|
||||
"description": "List of allowed values",
|
||||
"uniqueItems": true
|
||||
},
|
||||
"ontologies": {
|
||||
"type": "array",
|
||||
"description": "Ontology annotations (e.g., EDAM)",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"patternProperties": {
|
||||
"^[a-zA-Z]+$": {
|
||||
"type": "string",
|
||||
"format": "uri",
|
||||
"description": "Ontology URI"
|
||||
}
|
||||
}
|
||||
},
|
||||
"uniqueItems": true
|
||||
}
|
||||
},
|
||||
"required": ["type", "description"]
|
||||
}
|
||||
},
|
||||
"allOf": [
|
||||
{
|
||||
"if": {
|
||||
"properties": {
|
||||
"name": {
|
||||
"pattern": "^[a-z0-9][a-z0-9-]*/"
|
||||
}
|
||||
},
|
||||
"required": ["name"]
|
||||
},
|
||||
"then": {
|
||||
"required": ["name", "description", "version"],
|
||||
"properties": {
|
||||
"version": {
|
||||
"description": "Version is required for scoped/registry modules (scope/name format)"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"examples": [
|
||||
{
|
||||
"name": "fastqc",
|
||||
"description": "Run FastQC on sequenced reads",
|
||||
"keywords": ["quality control", "qc", "adapters", "fastq"],
|
||||
"tools": [
|
||||
{
|
||||
"fastqc": {
|
||||
"description": "FastQC gives general quality metrics about your reads.",
|
||||
"homepage": "https://www.bioinformatics.babraham.ac.uk/projects/fastqc/",
|
||||
"documentation": "https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/",
|
||||
"licence": ["GPL-2.0-only"],
|
||||
"identifier": "biotools:fastqc"
|
||||
}
|
||||
}
|
||||
],
|
||||
"input": [
|
||||
[
|
||||
{
|
||||
"name": "meta",
|
||||
"type": "map",
|
||||
"description": "Groovy Map containing sample information"
|
||||
},
|
||||
{
|
||||
"name": "reads",
|
||||
"type": "file",
|
||||
"description": "Input FastQ files",
|
||||
"ontologies": []
|
||||
}
|
||||
]
|
||||
],
|
||||
"output": [
|
||||
[
|
||||
{
|
||||
"type": "map",
|
||||
"description": "Sample information"
|
||||
},
|
||||
{
|
||||
"type": "file",
|
||||
"description": "FastQC report",
|
||||
"pattern": "*_{fastqc.html}",
|
||||
"ontologies": []
|
||||
}
|
||||
],
|
||||
{
|
||||
"name": "versions",
|
||||
"type": "file",
|
||||
"description": "File containing software versions",
|
||||
"pattern": "versions.yml"
|
||||
}
|
||||
],
|
||||
"authors": ["@drpatelh", "@ewels"],
|
||||
"maintainers": ["@drpatelh", "@ewels"]
|
||||
},
|
||||
{
|
||||
"name": "nf-core/bwa-align",
|
||||
"version": "1.2.4",
|
||||
"description": "Align reads using BWA-MEM algorithm",
|
||||
"keywords": ["alignment", "bwa", "mapping", "fastq", "bam"],
|
||||
"license": "MIT",
|
||||
"authors": ["@nf-core"],
|
||||
"maintainers": ["@nf-core"],
|
||||
"requires": {
|
||||
"nextflow": ">=24.04.0"
|
||||
},
|
||||
"tools": [
|
||||
{
|
||||
"bwa": {
|
||||
"description": "BWA aligner",
|
||||
"homepage": "http://bio-bwa.sourceforge.net/",
|
||||
"licence": ["GPL-3.0-or-later"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"samtools": {
|
||||
"description": "SAMtools",
|
||||
"homepage": "http://www.htslib.org/",
|
||||
"licence": ["MIT"]
|
||||
}
|
||||
}
|
||||
],
|
||||
"input": [
|
||||
[
|
||||
{
|
||||
"name": "meta",
|
||||
"type": "map",
|
||||
"description": "Sample metadata map"
|
||||
},
|
||||
{
|
||||
"name": "reads",
|
||||
"type": "file",
|
||||
"description": "Input FastQ files",
|
||||
"ontologies": [
|
||||
{ "edam": "http://edamontology.org/format_1930" }
|
||||
]
|
||||
}
|
||||
],
|
||||
{
|
||||
"name": "index",
|
||||
"type": "directory",
|
||||
"description": "BWA index directory"
|
||||
}
|
||||
],
|
||||
"output": [
|
||||
[
|
||||
{
|
||||
"type": "map",
|
||||
"description": "Sample metadata"
|
||||
},
|
||||
{
|
||||
"type": "file",
|
||||
"description": "Aligned BAM file",
|
||||
"pattern": "*.bam",
|
||||
"ontologies": [
|
||||
{ "edam": "http://edamontology.org/format_2572" }
|
||||
]
|
||||
}
|
||||
],
|
||||
{
|
||||
"name": "versions",
|
||||
"type": "file",
|
||||
"description": "Software versions"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
515
nextflow/build.gradle
Normal file
515
nextflow/build.gradle
Normal file
@@ -0,0 +1,515 @@
|
||||
/*
|
||||
* Copyright 2013-2026, Seqera Labs
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
plugins {
|
||||
id 'java'
|
||||
id 'idea'
|
||||
}
|
||||
|
||||
// Add ability to test with upcoming versions of Groovy
|
||||
def groovyVer = System.getenv('CI_GROOVY_VERSION')
|
||||
if (groovyVer) {
|
||||
def repo = groovyVer.startsWith('com.github.apache:') ? 'https://jitpack.io' : 'https://oss.jfrog.org/oss-snapshot-local/'
|
||||
logger.lifecycle "Overridden Groovy dependency to use $groovyVer - repository: $repo"
|
||||
allprojects {
|
||||
repositories {
|
||||
maven { url repo }
|
||||
}
|
||||
|
||||
configurations.all {
|
||||
resolutionStrategy.eachDependency { DependencyResolveDetails details ->
|
||||
if (details.requested.group == 'org.apache.groovy') {
|
||||
if( groovyVer.contains(':') )
|
||||
details.useTarget(groovyVer)
|
||||
else
|
||||
details.useVersion(groovyVer)
|
||||
println ">> Overriding $details.requested with version: $groovyVer"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def projects(String...args) {
|
||||
args.collect {project(it)}
|
||||
}
|
||||
|
||||
String gitVersion() {
|
||||
def p = new ProcessBuilder() .command('sh','-c','git rev-parse --short HEAD') .start()
|
||||
def r = p.waitFor()
|
||||
return r==0 ? p.text.trim() : '(unknown)'
|
||||
}
|
||||
|
||||
group = 'io.nextflow'
|
||||
version = rootProject.file('VERSION').text.trim()
|
||||
ext.commitId = gitVersion()
|
||||
|
||||
allprojects {
|
||||
apply plugin: 'java'
|
||||
apply plugin: 'java-test-fixtures'
|
||||
apply plugin: 'idea'
|
||||
apply plugin: 'groovy'
|
||||
apply plugin: 'java-library'
|
||||
apply plugin: 'jacoco'
|
||||
|
||||
java {
|
||||
// these settings apply to all jvm tooling, including groovy
|
||||
toolchain {
|
||||
languageVersion = JavaLanguageVersion.of(21)
|
||||
}
|
||||
sourceCompatibility = 17
|
||||
targetCompatibility = 17
|
||||
}
|
||||
|
||||
idea {
|
||||
module.inheritOutputDirs = true
|
||||
}
|
||||
|
||||
repositories {
|
||||
mavenCentral()
|
||||
maven { url 'https://repo.eclipse.org/content/groups/releases' }
|
||||
maven { url 'https://oss.sonatype.org/content/repositories/snapshots' }
|
||||
maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/releases" }
|
||||
maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/snapshots" }
|
||||
}
|
||||
|
||||
configurations {
|
||||
// see https://docs.gradle.org/4.1/userguide/dependency_management.html#sub:exclude_transitive_dependencies
|
||||
all*.exclude group: 'org.apache.groovy', module: 'groovy-all'
|
||||
all*.exclude group: 'org.apache.groovy', module: 'groovy-cli-picocli'
|
||||
// groovydoc libs
|
||||
groovyDoc.extendsFrom runtime
|
||||
}
|
||||
|
||||
dependencies {
|
||||
// see https://docs.gradle.org/4.1/userguide/dependency_management.html#sec:module_replacement
|
||||
modules {
|
||||
module("commons-logging:commons-logging") { replacedBy("org.slf4j:jcl-over-slf4j") }
|
||||
}
|
||||
|
||||
// JUnit Platform launcher required for Gradle 9.1+ when using useJUnitPlatform()
|
||||
testRuntimeOnly 'org.junit.platform:junit-platform-launcher:1.10.5'
|
||||
|
||||
// Documentation required libraries
|
||||
groovyDoc 'org.fusesource.jansi:jansi:2.4.0'
|
||||
groovyDoc "org.apache.groovy:groovy-groovydoc:4.0.31"
|
||||
groovyDoc "org.apache.groovy:groovy-ant:4.0.31"
|
||||
}
|
||||
|
||||
test {
|
||||
useJUnitPlatform()
|
||||
}
|
||||
|
||||
// this is required due to this IDEA bug
|
||||
// https://youtrack.jetbrains.com/issue/IDEA-129282
|
||||
sourceSets {
|
||||
main {
|
||||
output.resourcesDir = 'build/classes/main'
|
||||
}
|
||||
}
|
||||
|
||||
// Disable strict javadoc checks
|
||||
// See http://blog.joda.org/2014/02/turning-off-doclint-in-jdk-8-javadoc.html
|
||||
if (JavaVersion.current().isJava8Compatible()) {
|
||||
tasks.withType(Javadoc) {
|
||||
options.addStringOption('Xdoclint:none', '-quiet')
|
||||
}
|
||||
}
|
||||
|
||||
tasks.withType(Jar) {
|
||||
duplicatesStrategy = DuplicatesStrategy.INCLUDE
|
||||
}
|
||||
|
||||
// patched as described here
|
||||
// http://forums.gradle.org/gradle/topics/gradle_task_groovydoc_failing_with_noclassdeffounderror
|
||||
tasks.withType(Groovydoc) {
|
||||
groovyClasspath = project.configurations.groovyDoc
|
||||
includes = ["nextflow/**"]
|
||||
}
|
||||
|
||||
// Required to run tests on Java 9 and higher in compatibility mode
|
||||
tasks.withType(Test) {
|
||||
jvmArgs ([
|
||||
'--enable-preview',
|
||||
'--add-opens=java.base/java.lang=ALL-UNNAMED',
|
||||
'--add-opens=java.base/java.io=ALL-UNNAMED',
|
||||
'--add-opens=java.base/java.nio=ALL-UNNAMED',
|
||||
'--add-opens=java.base/java.nio.file.spi=ALL-UNNAMED',
|
||||
'--add-opens=java.base/java.net=ALL-UNNAMED',
|
||||
'--add-opens=java.base/java.util=ALL-UNNAMED',
|
||||
'--add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED',
|
||||
'--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED',
|
||||
'--add-opens=java.base/sun.nio.ch=ALL-UNNAMED',
|
||||
'--add-opens=java.base/sun.nio.fs=ALL-UNNAMED',
|
||||
'--add-opens=java.base/sun.net.www.protocol.http=ALL-UNNAMED',
|
||||
'--add-opens=java.base/sun.net.www.protocol.https=ALL-UNNAMED',
|
||||
'--add-opens=java.base/sun.net.www.protocol.ftp=ALL-UNNAMED',
|
||||
'--add-opens=java.base/sun.net.www.protocol.file=ALL-UNNAMED',
|
||||
'--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED',
|
||||
'--add-opens=java.base/jdk.internal.vm=ALL-UNNAMED',
|
||||
])
|
||||
}
|
||||
|
||||
/**
|
||||
* Code coverage with JaCoCo.
|
||||
* See: https://www.jacoco.org/; https://docs.gradle.org/current/userguide/jacoco_plugin.html
|
||||
*/
|
||||
// Code coverage report is always generated after tests run
|
||||
test { finalizedBy jacocoTestReport }
|
||||
jacocoTestReport {
|
||||
// Tests are required to run before generating the code coverage report
|
||||
dependsOn test
|
||||
|
||||
// Remove closure classes from the report, as they are already covered by the enclosing class coverage stats adding only noise.
|
||||
// See: https://stackoverflow.com/questions/39453696
|
||||
afterEvaluate {
|
||||
classDirectories.setFrom(files(classDirectories.files.collect { dir ->
|
||||
fileTree(dir: dir, excludes: ['**/*$*_closure*'])
|
||||
}))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// disable jar for root project
|
||||
jar.enabled = false
|
||||
|
||||
/*
|
||||
* Update the build timestamp in the source source file
|
||||
*/
|
||||
task buildInfo {
|
||||
// Always run this task - never consider it up-to-date
|
||||
outputs.upToDateWhen { false }
|
||||
|
||||
doLast {
|
||||
|
||||
def file0 = file('modules/nextflow/src/main/resources/META-INF/build-info.properties')
|
||||
def buildNum = 0
|
||||
|
||||
// Use GitHub Actions run number if available, otherwise increment local counter
|
||||
if (System.getenv('GITHUB_RUN_NUMBER')) {
|
||||
buildNum = System.getenv('GITHUB_RUN_NUMBER').toInteger()
|
||||
println "Using GitHub Actions run number: $buildNum"
|
||||
}
|
||||
|
||||
// -- update build-info file
|
||||
file0.text = """\
|
||||
build=${buildNum}
|
||||
version=${version}
|
||||
timestamp=${System.currentTimeMillis()}
|
||||
commitId=${project.property('commitId')}
|
||||
""".stripIndent()
|
||||
}}
|
||||
|
||||
/*
|
||||
* Update release information in nextflow wrapper, dockerfile, and plugin metadata.
|
||||
*
|
||||
* This task:
|
||||
* 1. Updates the NXF_VER version string in the nextflow launch script
|
||||
* 2. Updates the release version in docker/Dockerfile
|
||||
* 3. Generates plugins-info.txt with current plugin versions from VERSION files
|
||||
*
|
||||
* This task always runs to ensure all release artifacts are updated with current versions.
|
||||
*/
|
||||
task releaseInfo {
|
||||
dependsOn buildInfo
|
||||
// Always run this task - never consider it up-to-date
|
||||
outputs.upToDateWhen { false }
|
||||
|
||||
doLast {
|
||||
|
||||
// -- update 'nextflow' wrapper
|
||||
def file0 = file('nextflow')
|
||||
def src = file0.text
|
||||
src = src.replaceAll(/NXF_VER\=\$\{NXF_VER:-'.*'\}/, 'NXF_VER=\\${NXF_VER:-\'' + version + '\'}')
|
||||
file0.text = src
|
||||
|
||||
// -- update dockerfile
|
||||
file0 = file('docker/Dockerfile')
|
||||
src = file0.text
|
||||
src = src.replaceAll(/releases\/v[0-9a-zA-Z_\-\.]+\//, "releases/v$version/" as String)
|
||||
file0.text = src
|
||||
|
||||
// -- create plugins-info file
|
||||
def plugins = []
|
||||
new File(rootProject.rootDir, 'plugins')
|
||||
.eachDir { if(it.name.startsWith('nf-')) plugins << project(":plugins:${it.name}") }
|
||||
def meta = plugins.collect { "$it.name@$it.version" }
|
||||
file('modules/nextflow/src/main/resources/META-INF/plugins-info.txt').text = meta.toSorted().join('\n')
|
||||
}}
|
||||
|
||||
/*
|
||||
* Validate that plugins-info.txt matches plugin VERSION files and that build-info.properties
|
||||
* contains the correct build number and commit ID when running in GitHub Actions.
|
||||
*
|
||||
* This task ensures:
|
||||
* 1. All plugin versions in plugins-info.txt match their corresponding VERSION files
|
||||
* 2. The build number in build-info.properties matches GITHUB_RUN_NUMBER (in CI)
|
||||
* 3. The commit ID in build-info.properties matches GITHUB_SHA (in CI)
|
||||
*
|
||||
* This validation prevents releases with stale or mismatched metadata.
|
||||
*/
|
||||
task validatePluginVersions {
|
||||
dependsOn buildInfo
|
||||
inputs.file('modules/nextflow/src/main/resources/META-INF/plugins-info.txt')
|
||||
inputs.file('modules/nextflow/src/main/resources/META-INF/build-info.properties')
|
||||
inputs.files(fileTree('plugins') { include '*/VERSION' })
|
||||
|
||||
doLast {
|
||||
// Get expected versions from plugin projects
|
||||
def expected = []
|
||||
new File(rootProject.rootDir, 'plugins')
|
||||
.eachDir { if(it.name.startsWith('nf-')) expected << project(":plugins:${it.name}") }
|
||||
def expectedVersions = expected.collect { "$it.name@$it.version" }.toSorted()
|
||||
|
||||
// Get actual versions from plugins-info.txt
|
||||
def actualVersions = file('modules/nextflow/src/main/resources/META-INF/plugins-info.txt').readLines()
|
||||
|
||||
// Compare plugin versions
|
||||
if (expectedVersions != actualVersions) {
|
||||
def diffs = []
|
||||
expectedVersions.eachWithIndex { exp, i ->
|
||||
def act = actualVersions.size() > i ? actualVersions[i] : 'missing'
|
||||
if (exp != act) diffs << " expected: $exp, actual: $act"
|
||||
}
|
||||
throw new GradleException("Plugin version mismatch:\n${diffs.join('\n')}\nRun 'make assemble' to fix.")
|
||||
}
|
||||
|
||||
// Validate build-info.properties - require GitHub Actions environment variables
|
||||
if (!System.getenv('GITHUB_RUN_NUMBER')) {
|
||||
throw new GradleException("GITHUB_RUN_NUMBER environment variable is required")
|
||||
}
|
||||
if (!System.getenv('GITHUB_SHA')) {
|
||||
throw new GradleException("GITHUB_SHA environment variable is required")
|
||||
}
|
||||
|
||||
def buildInfoFile = file('modules/nextflow/src/main/resources/META-INF/build-info.properties')
|
||||
def props = new Properties()
|
||||
buildInfoFile.withInputStream { props.load(it) }
|
||||
|
||||
def actualBuild = props.getProperty('build')
|
||||
def expectedBuild = System.getenv('GITHUB_RUN_NUMBER')
|
||||
if (actualBuild != expectedBuild) {
|
||||
throw new GradleException("Build number mismatch: build-info.properties has '${actualBuild}' but GITHUB_RUN_NUMBER is '${expectedBuild}'. Run 'make assemble' to fix.")
|
||||
}
|
||||
|
||||
def actualCommit = props.getProperty('commitId')
|
||||
def expectedCommit = System.getenv('GITHUB_SHA').take(9) // GitHub SHA is full hash, we use short form
|
||||
if (actualCommit != expectedCommit) {
|
||||
throw new GradleException("Commit ID mismatch: build-info.properties has '${actualCommit}' but GITHUB_SHA is '${expectedCommit}'. Run 'make assemble' to fix.")
|
||||
}
|
||||
|
||||
println "✅ Build info validation passed: build=${actualBuild}, commitId=${actualCommit}"
|
||||
println "✅ Plugin version validation passed: all ${expected.size()} plugin versions match"
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* Compile sources and copies all libs to target directory
|
||||
*/
|
||||
task compile {
|
||||
dependsOn allprojects.classes
|
||||
}
|
||||
|
||||
def getRuntimeConfigs() {
|
||||
def names = subprojects
|
||||
.findAll { prj -> prj.name in ['nextflow','nf-commons','nf-httpfs','nf-lang','nf-lineage'] }
|
||||
.collect { it.name }
|
||||
|
||||
FileCollection result = null
|
||||
for( def it : names ) {
|
||||
def cfg = project(it).configurations.getByName('runtimeClasspath')
|
||||
if( result==null )
|
||||
result = cfg
|
||||
else
|
||||
result += cfg
|
||||
// this include the module actual jar file
|
||||
// note: migrating to gradle 7 does not work any more
|
||||
//result = result + cfg.getOutgoing().getArtifacts().getFiles()
|
||||
}
|
||||
return result?.files ?: []
|
||||
}
|
||||
|
||||
/*
|
||||
* Save the runtime classpath
|
||||
* NOTE: This task uses a provider to delay execution, but still triggers configuration
|
||||
* resolution when the provider is evaluated. While not ideal for Gradle 9.1's strict
|
||||
* configuration resolution requirements, this approach works in practice for our use case.
|
||||
*/
|
||||
task exportClasspath {
|
||||
dependsOn allprojects.jar
|
||||
|
||||
// Use provider to delay configuration resolution until task execution
|
||||
def configurationFiles = provider {
|
||||
def libs = []
|
||||
|
||||
// Resolve configurations during provider evaluation (not ideal but functional)
|
||||
['nextflow','nf-commons','nf-httpfs','nf-lang','nf-lineage'].each { moduleName ->
|
||||
def moduleProject = project(":$moduleName")
|
||||
def cfg = moduleProject.configurations.getByName('runtimeClasspath')
|
||||
libs.addAll(cfg.files.collect { it.canonicalPath })
|
||||
}
|
||||
|
||||
// Add module jars
|
||||
['nextflow','nf-commons','nf-httpfs','nf-lang','nf-lineage'].each {
|
||||
libs << file("modules/$it/build/libs/${it}-${version}.jar").canonicalPath
|
||||
}
|
||||
|
||||
return libs.unique()
|
||||
}
|
||||
|
||||
inputs.files(configurationFiles)
|
||||
outputs.file('.launch.classpath')
|
||||
|
||||
doLast {
|
||||
def libs = configurationFiles.get()
|
||||
file('.launch.classpath').text = libs.join(':')
|
||||
}
|
||||
}
|
||||
|
||||
ext.nexusUsername = project.findProperty('nexusUsername') ?: System.getenv('AWS_ACCESS_KEY_ID')
|
||||
ext.nexusPassword = project.findProperty('nexusPassword') ?: System.getenv('AWS_SECRET_ACCESS_KEY')
|
||||
ext.nexusFullName = project.findProperty('nexusFullName')
|
||||
ext.nexusEmail = project.findProperty('nexusEmail')
|
||||
|
||||
// `signing.keyId` property needs to be defined in the `gradle.properties` file
|
||||
ext.enableSignArchives = project.findProperty('signing.keyId')
|
||||
|
||||
ext.coreProjects = projects( ':nextflow', ':nf-commons', ':nf-httpfs', ':nf-lang', ':nf-lineage' )
|
||||
|
||||
configure(coreProjects) {
|
||||
group = 'io.nextflow'
|
||||
version = rootProject.file('VERSION').text.trim()
|
||||
}
|
||||
|
||||
/*
|
||||
* Maven central deployment
|
||||
* http://central.sonatype.org/pages/gradle.html
|
||||
*/
|
||||
configure(coreProjects) {
|
||||
apply plugin: 'maven-publish'
|
||||
apply plugin: 'signing'
|
||||
|
||||
task javadocJar(type: Jar) {
|
||||
archiveClassifier = 'javadoc'
|
||||
from configurations.groovyDoc
|
||||
}
|
||||
|
||||
task sourcesJar(type: Jar) {
|
||||
archiveClassifier = 'sources'
|
||||
from sourceSets.main.allSource
|
||||
}
|
||||
|
||||
publishing {
|
||||
publications {
|
||||
mavenJava(MavenPublication) {
|
||||
suppressPomMetadataWarningsFor('testFixturesApiElements')
|
||||
suppressPomMetadataWarningsFor('testFixturesRuntimeElements')
|
||||
from components.java
|
||||
versionMapping {
|
||||
usage('java-api') {
|
||||
fromResolutionOf('runtimeClasspath')
|
||||
}
|
||||
usage('java-runtime') {
|
||||
fromResolutionResult()
|
||||
}
|
||||
}
|
||||
pom {
|
||||
name = 'Nextflow'
|
||||
description = 'A DSL modelled around the UNIX pipe concept, that simplifies writing parallel and scalable pipelines in a portable manner'
|
||||
url = 'http://www.nextflow.io'
|
||||
licenses {
|
||||
license {
|
||||
name = 'The Apache License, Version 2.0'
|
||||
url = 'http://www.apache.org/licenses/LICENSE-2.0.txt'
|
||||
}
|
||||
}
|
||||
developers {
|
||||
developer {
|
||||
id = nexusUsername
|
||||
name = nexusFullName
|
||||
email = nexusEmail
|
||||
}
|
||||
}
|
||||
scm {
|
||||
connection = 'scm:git:https://github.com/nextflow-io/nextflow'
|
||||
developerConnection = 'scm:git:git@github.com:nextflow-io/nextflow.git'
|
||||
url = 'https://github.com/nextflow-io/nextflow'
|
||||
}
|
||||
}
|
||||
|
||||
artifact sourcesJar
|
||||
artifact javadocJar
|
||||
}
|
||||
}
|
||||
|
||||
repositories {
|
||||
maven {
|
||||
name = 'Seqera'
|
||||
// change URLs to point to your repos, e.g. http://my.org/repo
|
||||
def releasesRepoUrl = "s3://maven.seqera.io/releases/"
|
||||
def snapshotsRepoUrl = "s3://maven.seqera.io/snapshots/"
|
||||
url = version.endsWith('SNAPSHOT') ? snapshotsRepoUrl : releasesRepoUrl
|
||||
credentials(AwsCredentials) {
|
||||
accessKey nexusUsername
|
||||
secretKey nexusPassword
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
signing {
|
||||
required { enableSignArchives }
|
||||
sign publishing.publications.mavenJava
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
|
||||
String bytesToHex(byte[] bytes) {
|
||||
StringBuffer result = new StringBuffer();
|
||||
for (byte byt : bytes) result.append(Integer.toString((byt & 0xff) + 0x100, 16).substring(1));
|
||||
return result.toString();
|
||||
}
|
||||
|
||||
task makeDigest { doLast {
|
||||
byte[] digest
|
||||
String str = file('nextflow').text
|
||||
// create sha1
|
||||
digest = java.security.MessageDigest.getInstance("SHA1").digest(str.getBytes())
|
||||
file('nextflow.sha1').text = new BigInteger(1, digest).toString(16) + '\n'
|
||||
// create sha-256
|
||||
digest = java.security.MessageDigest.getInstance("SHA-256").digest(str.getBytes())
|
||||
file('nextflow.sha256').text = bytesToHex(digest) + '\n'
|
||||
// create md5
|
||||
digest = java.security.MessageDigest.getInstance("MD5").digest(str.getBytes())
|
||||
file('nextflow.md5').text = bytesToHex(digest) + '\n'
|
||||
}}
|
||||
|
||||
// Make releaseInfo task automatically run makeDigest after updating versions
|
||||
releaseInfo.finalizedBy makeDigest
|
||||
|
||||
|
||||
task upload {
|
||||
dependsOn compile
|
||||
dependsOn coreProjects.publish
|
||||
dependsOn validatePluginVersions
|
||||
}
|
||||
|
||||
|
||||
if( System.env.BUILD_PACK ) {
|
||||
apply from: 'packing.gradle'
|
||||
}
|
||||
|
||||
4749
nextflow/changelog.txt
Normal file
4749
nextflow/changelog.txt
Normal file
File diff suppressed because it is too large
Load Diff
18
nextflow/compile.sh
Executable file
18
nextflow/compile.sh
Executable file
@@ -0,0 +1,18 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Copyright 2013-2026, Seqera Labs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
./gradlew -q compile exportClasspath
|
||||
351
nextflow/config/codenarc/codenarc.groovy
Normal file
351
nextflow/config/codenarc/codenarc.groovy
Normal file
@@ -0,0 +1,351 @@
|
||||
/*
|
||||
* Copyright 2013-2026, Seqera Labs
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
ruleset {
|
||||
|
||||
description '''
|
||||
A Sample Groovy RuleSet containing all CodeNarc Rules, grouped by category.
|
||||
You can use this as a template for your own custom RuleSet.
|
||||
Just delete the rules that you don't want to include.
|
||||
'''
|
||||
|
||||
// rulesets/basic.xml
|
||||
AssertWithinFinallyBlock
|
||||
AssignmentInConditional
|
||||
BigDecimalInstantiation
|
||||
BitwiseOperatorInConditional
|
||||
BooleanGetBoolean
|
||||
BrokenNullCheck
|
||||
BrokenOddnessCheck
|
||||
ClassForName
|
||||
ComparisonOfTwoConstants
|
||||
ComparisonWithSelf
|
||||
ConstantAssertExpression
|
||||
ConstantIfExpression
|
||||
ConstantTernaryExpression
|
||||
DeadCode
|
||||
DoubleNegative
|
||||
DuplicateCaseStatement
|
||||
DuplicateMapKey
|
||||
DuplicateSetValue
|
||||
EmptyCatchBlock
|
||||
EmptyElseBlock
|
||||
EmptyFinallyBlock
|
||||
EmptyForStatement
|
||||
EmptyIfStatement
|
||||
EmptyInstanceInitializer
|
||||
EmptyMethod
|
||||
EmptyStaticInitializer
|
||||
EmptySwitchStatement
|
||||
EmptySynchronizedStatement
|
||||
EmptyTryBlock
|
||||
EmptyWhileStatement
|
||||
EqualsAndHashCode
|
||||
EqualsOverloaded
|
||||
ExplicitGarbageCollection
|
||||
ForLoopShouldBeWhileLoop
|
||||
HardCodedWindowsFileSeparator
|
||||
HardCodedWindowsRootDirectory
|
||||
IntegerGetInteger
|
||||
RandomDoubleCoercedToZero
|
||||
RemoveAllOnSelf
|
||||
ReturnFromFinallyBlock
|
||||
ThrowExceptionFromFinallyBlock
|
||||
|
||||
// rulesets/braces.xml
|
||||
//ElseBlockBraces
|
||||
ForStatementBraces
|
||||
//IfStatementBraces
|
||||
//WhileStatementBraces
|
||||
|
||||
// rulesets/concurrency.xml
|
||||
BusyWait
|
||||
DoubleCheckedLocking
|
||||
InconsistentPropertyLocking
|
||||
InconsistentPropertySynchronization
|
||||
NestedSynchronization
|
||||
StaticCalendarField
|
||||
StaticConnection
|
||||
StaticDateFormatField
|
||||
StaticMatcherField
|
||||
StaticSimpleDateFormatField
|
||||
SynchronizedMethod
|
||||
SynchronizedOnBoxedPrimitive
|
||||
SynchronizedOnGetClass
|
||||
SynchronizedOnReentrantLock
|
||||
SynchronizedOnString
|
||||
SynchronizedOnThis
|
||||
SynchronizedReadObjectMethod
|
||||
SystemRunFinalizersOnExit
|
||||
ThreadGroup
|
||||
ThreadLocalNotStaticFinal
|
||||
ThreadYield
|
||||
UseOfNotifyMethod
|
||||
VolatileArrayField
|
||||
VolatileLongOrDoubleField
|
||||
WaitOutsideOfWhileLoop
|
||||
|
||||
// rulesets/convention.xml
|
||||
ConfusingTernary
|
||||
CouldBeElvis
|
||||
HashtableIsObsolete
|
||||
IfStatementCouldBeTernary
|
||||
InvertedIfElse
|
||||
LongLiteralWithLowerCaseL
|
||||
ParameterReassignment
|
||||
TernaryCouldBeElvis
|
||||
VectorIsObsolete
|
||||
|
||||
// rulesets/design.xml
|
||||
AbstractClassWithPublicConstructor
|
||||
AbstractClassWithoutAbstractMethod
|
||||
BooleanMethodReturnsNull
|
||||
BuilderMethodWithSideEffects
|
||||
CloneableWithoutClone
|
||||
CloseWithoutCloseable
|
||||
CompareToWithoutComparable
|
||||
ConstantsOnlyInterface
|
||||
EmptyMethodInAbstractClass
|
||||
FinalClassWithProtectedMember
|
||||
ImplementationAsType
|
||||
PrivateFieldCouldBeFinal
|
||||
PublicInstanceField
|
||||
ReturnsNullInsteadOfEmptyArray
|
||||
ReturnsNullInsteadOfEmptyCollection
|
||||
SimpleDateFormatMissingLocale
|
||||
StatelessSingleton
|
||||
|
||||
// rulesets/dry.xml
|
||||
DuplicateListLiteral
|
||||
DuplicateMapLiteral
|
||||
DuplicateNumberLiteral
|
||||
DuplicateStringLiteral
|
||||
|
||||
// rulesets/exceptions.xml
|
||||
CatchArrayIndexOutOfBoundsException
|
||||
CatchError
|
||||
CatchException
|
||||
CatchIllegalMonitorStateException
|
||||
CatchIndexOutOfBoundsException
|
||||
CatchNullPointerException
|
||||
CatchRuntimeException
|
||||
CatchThrowable
|
||||
ConfusingClassNamedException
|
||||
ExceptionExtendsError
|
||||
ExceptionNotThrown
|
||||
MissingNewInThrowStatement
|
||||
ReturnNullFromCatchBlock
|
||||
SwallowThreadDeath
|
||||
ThrowError
|
||||
ThrowException
|
||||
ThrowNullPointerException
|
||||
ThrowRuntimeException
|
||||
ThrowThrowable
|
||||
|
||||
// rulesets/generic.xml
|
||||
IllegalClassReference
|
||||
IllegalPackageReference
|
||||
IllegalRegex
|
||||
RequiredRegex
|
||||
RequiredString
|
||||
StatelessClass
|
||||
|
||||
// rulesets/grails.xml
|
||||
GrailsDomainHasEquals
|
||||
GrailsDomainHasToString
|
||||
GrailsDuplicateConstraint
|
||||
GrailsDuplicateMapping
|
||||
GrailsPublicControllerMethod
|
||||
GrailsServletContextReference
|
||||
GrailsSessionReference // DEPRECATED
|
||||
GrailsStatelessService
|
||||
|
||||
// rulesets/groovyism.xml
|
||||
AssignCollectionSort
|
||||
AssignCollectionUnique
|
||||
ClosureAsLastMethodParameter
|
||||
CollectAllIsDeprecated
|
||||
ConfusingMultipleReturns
|
||||
ExplicitArrayListInstantiation
|
||||
ExplicitCallToAndMethod
|
||||
ExplicitCallToCompareToMethod
|
||||
ExplicitCallToDivMethod
|
||||
ExplicitCallToEqualsMethod
|
||||
ExplicitCallToGetAtMethod
|
||||
ExplicitCallToLeftShiftMethod
|
||||
ExplicitCallToMinusMethod
|
||||
ExplicitCallToModMethod
|
||||
ExplicitCallToMultiplyMethod
|
||||
ExplicitCallToOrMethod
|
||||
ExplicitCallToPlusMethod
|
||||
ExplicitCallToPowerMethod
|
||||
ExplicitCallToRightShiftMethod
|
||||
ExplicitCallToXorMethod
|
||||
ExplicitHashMapInstantiation
|
||||
ExplicitHashSetInstantiation
|
||||
ExplicitLinkedHashMapInstantiation
|
||||
ExplicitLinkedListInstantiation
|
||||
ExplicitStackInstantiation
|
||||
ExplicitTreeSetInstantiation
|
||||
GStringAsMapKey
|
||||
GetterMethodCouldBeProperty
|
||||
GroovyLangImmutable
|
||||
UseCollectMany
|
||||
UseCollectNested
|
||||
|
||||
// rulesets/imports.xml
|
||||
DuplicateImport
|
||||
ImportFromSamePackage
|
||||
ImportFromSunPackages
|
||||
MisorderedStaticImports
|
||||
UnnecessaryGroovyImport
|
||||
UnusedImport
|
||||
|
||||
// rulesets/jdbc.xml
|
||||
DirectConnectionManagement
|
||||
JdbcConnectionReference
|
||||
JdbcResultSetReference
|
||||
JdbcStatementReference
|
||||
|
||||
// rulesets/junit.xml
|
||||
ChainedTest
|
||||
CoupledTestCase
|
||||
JUnitAssertAlwaysFails
|
||||
JUnitAssertAlwaysSucceeds
|
||||
JUnitFailWithoutMessage
|
||||
JUnitLostTest
|
||||
JUnitPublicNonTestMethod
|
||||
JUnitSetUpCallsSuper
|
||||
JUnitStyleAssertions
|
||||
JUnitTearDownCallsSuper
|
||||
JUnitTestMethodWithoutAssert
|
||||
JUnitUnnecessarySetUp
|
||||
JUnitUnnecessaryTearDown
|
||||
JUnitUnnecessaryThrowsException
|
||||
SpockIgnoreRestUsed
|
||||
UnnecessaryFail
|
||||
UseAssertEqualsInsteadOfAssertTrue
|
||||
UseAssertFalseInsteadOfNegation
|
||||
UseAssertNullInsteadOfAssertEquals
|
||||
UseAssertSameInsteadOfAssertTrue
|
||||
UseAssertTrueInsteadOfAssertEquals
|
||||
UseAssertTrueInsteadOfNegation
|
||||
|
||||
// rulesets/logging.xml
|
||||
LoggerForDifferentClass
|
||||
LoggerWithWrongModifiers
|
||||
LoggingSwallowsStacktrace
|
||||
MultipleLoggers
|
||||
PrintStackTrace
|
||||
Println
|
||||
SystemErrPrint
|
||||
SystemOutPrint
|
||||
|
||||
// rulesets/naming.xml
|
||||
AbstractClassName
|
||||
ClassName
|
||||
ConfusingMethodName
|
||||
FactoryMethodName
|
||||
FieldName
|
||||
InterfaceName
|
||||
MethodName
|
||||
ObjectOverrideMisspelledMethodName
|
||||
PackageName
|
||||
ParameterName
|
||||
PropertyName
|
||||
//VariableName
|
||||
|
||||
// rulesets/security.xml
|
||||
FileCreateTempFile
|
||||
InsecureRandom
|
||||
NonFinalPublicField
|
||||
NonFinalSubclassOfSensitiveInterface
|
||||
ObjectFinalize
|
||||
PublicFinalizeMethod
|
||||
SystemExit
|
||||
UnsafeArrayDeclaration
|
||||
|
||||
// rulesets/serialization.xml
|
||||
SerialPersistentFields
|
||||
SerialVersionUID
|
||||
SerializableClassMustDefineSerialVersionUID
|
||||
|
||||
// rulesets/size.xml
|
||||
AbcComplexity // DEPRECATED: Use the AbcMetric rule instead. Requires the GMetrics jar
|
||||
AbcMetric // Requires the GMetrics jar
|
||||
ClassSize
|
||||
CrapMetric // Requires the GMetrics jar and a Cobertura coverage file
|
||||
CyclomaticComplexity // Requires the GMetrics jar
|
||||
MethodCount
|
||||
MethodSize
|
||||
NestedBlockDepth
|
||||
|
||||
// rulesets/unnecessary.xml
|
||||
AddEmptyString
|
||||
ConsecutiveLiteralAppends
|
||||
ConsecutiveStringConcatenation
|
||||
UnnecessaryBigDecimalInstantiation
|
||||
UnnecessaryBigIntegerInstantiation
|
||||
UnnecessaryBooleanExpression
|
||||
UnnecessaryBooleanInstantiation
|
||||
UnnecessaryCallForLastElement
|
||||
UnnecessaryCallToSubstring
|
||||
UnnecessaryCatchBlock
|
||||
UnnecessaryCollectCall
|
||||
UnnecessaryCollectionCall
|
||||
UnnecessaryConstructor
|
||||
UnnecessaryDefInFieldDeclaration
|
||||
//UnnecessaryDefInMethodDeclaration
|
||||
UnnecessaryDefInVariableDeclaration
|
||||
UnnecessaryDotClass
|
||||
UnnecessaryDoubleInstantiation
|
||||
UnnecessaryElseStatement
|
||||
UnnecessaryFinalOnPrivateMethod
|
||||
UnnecessaryFloatInstantiation
|
||||
//UnnecessaryGString
|
||||
//UnnecessaryGetter
|
||||
UnnecessaryIfStatement
|
||||
UnnecessaryInstanceOfCheck
|
||||
UnnecessaryInstantiationToGetClass
|
||||
UnnecessaryIntegerInstantiation
|
||||
UnnecessaryLongInstantiation
|
||||
UnnecessaryModOne
|
||||
UnnecessaryNullCheck
|
||||
UnnecessaryNullCheckBeforeInstanceOf
|
||||
UnnecessaryObjectReferences
|
||||
UnnecessaryOverridingMethod
|
||||
UnnecessaryPackageReference
|
||||
UnnecessaryParenthesesForMethodCallWithClosure
|
||||
UnnecessaryPublicModifier
|
||||
UnnecessaryReturnKeyword
|
||||
UnnecessarySelfAssignment
|
||||
UnnecessarySemicolon
|
||||
UnnecessaryStringInstantiation
|
||||
UnnecessarySubstring
|
||||
UnnecessaryTernaryExpression
|
||||
UnnecessaryTransientModifier
|
||||
|
||||
// rulesets/unused.xml
|
||||
UnusedArray
|
||||
UnusedMethodParameter
|
||||
UnusedObject
|
||||
UnusedPrivateField
|
||||
UnusedPrivateMethod
|
||||
UnusedPrivateMethodParameter
|
||||
UnusedVariable
|
||||
|
||||
|
||||
}
|
||||
24
nextflow/config/codenarc/codenarc.xml
Normal file
24
nextflow/config/codenarc/codenarc.xml
Normal file
@@ -0,0 +1,24 @@
|
||||
<!--
|
||||
~ Copyright 2013-2026, Seqera Labs
|
||||
~
|
||||
~ Licensed under the Apache License, Version 2.0 (the "License");
|
||||
~ you may not use this file except in compliance with the License.
|
||||
~ You may obtain a copy of the License at
|
||||
~
|
||||
~ http://www.apache.org/licenses/LICENSE-2.0
|
||||
~
|
||||
~ Unless required by applicable law or agreed to in writing, software
|
||||
~ distributed under the License is distributed on an "AS IS" BASIS,
|
||||
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
~ See the License for the specific language governing permissions and
|
||||
~ limitations under the License.
|
||||
-->
|
||||
|
||||
<ruleset xmlns="http://codenarc.org/ruleset/1.0"
|
||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://codenarc.org/ruleset/1.0 http://codenarc.org/ruleset-schema.xsd"
|
||||
xsi:noNamespaceSchemaLocation="http://codenarc.org/ruleset-schema.xsd">
|
||||
|
||||
<ruleset-ref path='file:./config/codenarc/codenarc.groovy'/>
|
||||
|
||||
</ruleset>
|
||||
18
nextflow/console.sh
Executable file
18
nextflow/console.sh
Executable file
@@ -0,0 +1,18 @@
|
||||
#
|
||||
# Copyright 2013-2026, Seqera Labs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
export MAIN_CLASS='nextflow.ui.console.Nextflow'
|
||||
source ./launch.sh
|
||||
19
nextflow/docker/Dockerfile
Normal file
19
nextflow/docker/Dockerfile
Normal file
@@ -0,0 +1,19 @@
|
||||
FROM amazoncorretto:21-al2023
|
||||
RUN yum install -y procps-ng shadow-utils which
|
||||
|
||||
ENV NXF_HOME=/.nextflow
|
||||
ARG TARGETPLATFORM=linux/amd64
|
||||
|
||||
# copy docker client
|
||||
COPY dist/${TARGETPLATFORM}/docker /usr/local/bin/docker
|
||||
COPY entry.sh /usr/local/bin/entry.sh
|
||||
COPY nextflow /usr/local/bin/nextflow
|
||||
|
||||
# download runtime
|
||||
RUN mkdir /.nextflow \
|
||||
&& chmod 755 /usr/local/bin/nextflow \
|
||||
&& chmod 755 /usr/local/bin/entry.sh \
|
||||
&& nextflow info
|
||||
|
||||
# define the entry point
|
||||
ENTRYPOINT ["/usr/local/bin/entry.sh"]
|
||||
45
nextflow/docker/Makefile
Normal file
45
nextflow/docker/Makefile
Normal file
@@ -0,0 +1,45 @@
|
||||
#
|
||||
# Copyright 2013-2024, Seqera Labs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
version ?= none
|
||||
|
||||
build: dist/docker/amd64
|
||||
cp ../nextflow .
|
||||
docker buildx build --platform linux/amd64 --output=type=docker --progress=plain --tag nextflow/nextflow:${version} --build-arg TARGETPLATFORM=linux/amd64 .
|
||||
|
||||
build-arm: dist/docker/arm64
|
||||
cp ../nextflow .
|
||||
docker buildx build --platform linux/arm64 --output=type=docker --progress=plain --tag nextflow/nextflow:${version} --build-arg TARGETPLATFORM=linux/arm64 .
|
||||
|
||||
release: build
|
||||
docker push nextflow/nextflow:${version}
|
||||
#
|
||||
docker tag nextflow/nextflow:${version} public.cr.seqera.io/nextflow/nextflow:${version}
|
||||
docker push public.cr.seqera.io/nextflow/nextflow:${version}
|
||||
|
||||
#Static builds can now be found at:
|
||||
#
|
||||
# Linux: https://download.docker.com/linux/static
|
||||
# MacOS: https://download.docker.com/mac/static
|
||||
# Windows: https://download.docker.com/win/static
|
||||
|
||||
dist/docker/amd64:
|
||||
mkdir -p dist/linux/amd64
|
||||
curl -fsSL https://download.docker.com/linux/static/stable/x86_64/docker-19.03.15.tgz | tar --strip-components=1 -xvz -C dist/linux/amd64
|
||||
|
||||
dist/docker/arm64:
|
||||
mkdir -p dist/linux/arm64
|
||||
curl -fsSL https://download.docker.com/linux/static/stable/aarch64/docker-19.03.15.tgz | tar --strip-components=1 -xvz -C dist/linux/arm64
|
||||
|
||||
62
nextflow/docker/entry.sh
Executable file
62
nextflow/docker/entry.sh
Executable file
@@ -0,0 +1,62 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Copyright 2013-2026, Seqera Labs
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
#
|
||||
# This script acts as a pass-through container entry point. Its main role
|
||||
# is to create a user able to execute docker commands from the container
|
||||
# connecting to the host docker socket at runtime.
|
||||
#
|
||||
# The invoker needs to pass the user ID using the variable NXF_USRMAP.
|
||||
# When this variable is defined, it creates a new user in the container
|
||||
# with such ID and adds it to the `docker` group, then assigns the docker
|
||||
# socket file ownership to that user.
|
||||
#
|
||||
# Finally it switches the `nextflow` user using the `su` command and
|
||||
# executes the original target command line.
|
||||
#
|
||||
# authors:
|
||||
# Paolo Di Tommaso
|
||||
# Emilio Palumbo
|
||||
#
|
||||
|
||||
# enable debugging
|
||||
[[ "$NXF_DEBUG_ENTRY" ]] && set -x
|
||||
|
||||
# wrap cli args with single quote to avoid wildcard expansion
|
||||
cli=''; for x in "$@"; do cli+="'$x' "; done
|
||||
|
||||
# the NXF_USRMAP hold the user ID in the host environment
|
||||
if [[ "$NXF_USRMAP" ]]; then
|
||||
# create a `nextflow` user with the provided ID
|
||||
groupadd docker
|
||||
useradd -u "$NXF_USRMAP" -G docker -s /bin/bash nextflow
|
||||
|
||||
# then change the docker socket ownership to `nextflow` user
|
||||
# and change the $NXF_HOME ownership to `nextflow` user
|
||||
chown nextflow /var/run/docker.sock
|
||||
chown -R nextflow /.nextflow
|
||||
|
||||
# finally run the target command with `nextflow` user
|
||||
su nextflow << EOF
|
||||
[[ "$NXF_DEBUG_ENTRY" ]] && set -x
|
||||
exec bash -c "$cli"
|
||||
EOF
|
||||
|
||||
# otherwise just execute the command
|
||||
else
|
||||
exec bash -c "$cli"
|
||||
fi
|
||||
8
nextflow/docker/flux/README.md
Normal file
8
nextflow/docker/flux/README.md
Normal file
@@ -0,0 +1,8 @@
|
||||
# Flux Container Environment
|
||||
|
||||
In this directory we include a [Dockerfile](.devcontainer/Dockerfile)
|
||||
along with a [VSCode Developer Container](https://code.visualstudio.com/docs/devcontainers/containers)
|
||||
environment that you can put at the root of the project to be provided with a Flux agent and the dependencies
|
||||
needed to build Nextflow. See the [documentation](https://www.nextflow.io/docs/latest/flux.html)
|
||||
for the latest for how to use them.
|
||||
|
||||
53
nextflow/docs/.wci.yml
Normal file
53
nextflow/docs/.wci.yml
Normal file
@@ -0,0 +1,53 @@
|
||||
---
|
||||
# Metadata parsed by the workflow community initiative, https://workflows.community/systems
|
||||
# Registered at https://github.com/workflowscommunity/workflowscommunity.github.io/blob/main/_data/workflow_systems.yml
|
||||
name: Nextflow
|
||||
|
||||
headline: A DSL for data-driven computational pipelines
|
||||
|
||||
description: |
|
||||
Nextflow is an open-source workflow orchestrator that simplifies writing and deploying containerised data pipelines in a portable and scalable manner. It supports deploying workflows on a variety of execution platforms including local computers, HPC batch schedulers, Kubernetes and cloud platforms such as AWS, Azure and Google Cloud.
|
||||
|
||||
language: Groovy
|
||||
|
||||
documentation:
|
||||
general: http://docs.nextflow.io
|
||||
installation: https://www.nextflow.io/docs/latest/getstarted.html
|
||||
tutorial: https://training.nextflow.io
|
||||
|
||||
social:
|
||||
twitter: nextflowio
|
||||
youtube: https://www.youtube.com/c/Nextflow
|
||||
|
||||
execution_environment:
|
||||
interfaces:
|
||||
- GitHub
|
||||
- GitLab
|
||||
- BitBucket
|
||||
resource_managers:
|
||||
- Slurm
|
||||
- LSF
|
||||
- PBS/Torque
|
||||
- PBS Pro
|
||||
- SGE
|
||||
- Moab
|
||||
- HTCondor
|
||||
- Bridge
|
||||
- HyperQueue
|
||||
- GLS
|
||||
- NQSII
|
||||
- OAR
|
||||
- Local
|
||||
- AWS
|
||||
- GCP
|
||||
- Azure
|
||||
- Kubernetes
|
||||
transfer_protocols:
|
||||
- HTTP
|
||||
- HTTPS
|
||||
- FTP
|
||||
- Amazon S3
|
||||
- Google Storage
|
||||
- Azure Storage
|
||||
- Git+HTTPS
|
||||
|
||||
14
nextflow/docs/Dockerfile
Normal file
14
nextflow/docs/Dockerfile
Normal file
@@ -0,0 +1,14 @@
|
||||
FROM mambaorg/micromamba:1.5.8
|
||||
|
||||
RUN micromamba install --yes --name base --channel conda-forge \
|
||||
make=4.3 \
|
||||
python=3.7 \
|
||||
conda-forge:git=2.45.0 \
|
||||
&& \
|
||||
micromamba clean --all --yes
|
||||
|
||||
COPY requirements.txt .
|
||||
|
||||
RUN eval "$(micromamba shell hook --shell=bash)" && \
|
||||
micromamba activate && \
|
||||
pip install -r requirements.txt
|
||||
428
nextflow/docs/LICENCE.txt
Normal file
428
nextflow/docs/LICENCE.txt
Normal file
@@ -0,0 +1,428 @@
|
||||
Attribution-ShareAlike 4.0 International
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons Corporation ("Creative Commons") is not a law firm and
|
||||
does not provide legal services or legal advice. Distribution of
|
||||
Creative Commons public licenses does not create a lawyer-client or
|
||||
other relationship. Creative Commons makes its licenses and related
|
||||
information available on an "as-is" basis. Creative Commons gives no
|
||||
warranties regarding its licenses, any material licensed under their
|
||||
terms and conditions, or any related information. Creative Commons
|
||||
disclaims all liability for damages resulting from their use to the
|
||||
fullest extent possible.
|
||||
|
||||
Using Creative Commons Public Licenses
|
||||
|
||||
Creative Commons public licenses provide a standard set of terms and
|
||||
conditions that creators and other rights holders may use to share
|
||||
original works of authorship and other material subject to copyright
|
||||
and certain other rights specified in the public license below. The
|
||||
following considerations are for informational purposes only, are not
|
||||
exhaustive, and do not form part of our licenses.
|
||||
|
||||
Considerations for licensors: Our public licenses are
|
||||
intended for use by those authorized to give the public
|
||||
permission to use material in ways otherwise restricted by
|
||||
copyright and certain other rights. Our licenses are
|
||||
irrevocable. Licensors should read and understand the terms
|
||||
and conditions of the license they choose before applying it.
|
||||
Licensors should also secure all rights necessary before
|
||||
applying our licenses so that the public can reuse the
|
||||
material as expected. Licensors should clearly mark any
|
||||
material not subject to the license. This includes other CC-
|
||||
licensed material, or material used under an exception or
|
||||
limitation to copyright. More considerations for licensors:
|
||||
wiki.creativecommons.org/Considerations_for_licensors
|
||||
|
||||
Considerations for the public: By using one of our public
|
||||
licenses, a licensor grants the public permission to use the
|
||||
licensed material under specified terms and conditions. If
|
||||
the licensor's permission is not necessary for any reason--for
|
||||
example, because of any applicable exception or limitation to
|
||||
copyright--then that use is not regulated by the license. Our
|
||||
licenses grant only permissions under copyright and certain
|
||||
other rights that a licensor has authority to grant. Use of
|
||||
the licensed material may still be restricted for other
|
||||
reasons, including because others have copyright or other
|
||||
rights in the material. A licensor may make special requests,
|
||||
such as asking that all changes be marked or described.
|
||||
Although not required by our licenses, you are encouraged to
|
||||
respect those requests where reasonable. More considerations
|
||||
for the public:
|
||||
wiki.creativecommons.org/Considerations_for_licensees
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons Attribution-ShareAlike 4.0 International Public
|
||||
License
|
||||
|
||||
By exercising the Licensed Rights (defined below), You accept and agree
|
||||
to be bound by the terms and conditions of this Creative Commons
|
||||
Attribution-ShareAlike 4.0 International Public License ("Public
|
||||
License"). To the extent this Public License may be interpreted as a
|
||||
contract, You are granted the Licensed Rights in consideration of Your
|
||||
acceptance of these terms and conditions, and the Licensor grants You
|
||||
such rights in consideration of benefits the Licensor receives from
|
||||
making the Licensed Material available under these terms and
|
||||
conditions.
|
||||
|
||||
|
||||
Section 1 -- Definitions.
|
||||
|
||||
a. Adapted Material means material subject to Copyright and Similar
|
||||
Rights that is derived from or based upon the Licensed Material
|
||||
and in which the Licensed Material is translated, altered,
|
||||
arranged, transformed, or otherwise modified in a manner requiring
|
||||
permission under the Copyright and Similar Rights held by the
|
||||
Licensor. For purposes of this Public License, where the Licensed
|
||||
Material is a musical work, performance, or sound recording,
|
||||
Adapted Material is always produced where the Licensed Material is
|
||||
synched in timed relation with a moving image.
|
||||
|
||||
b. Adapter's License means the license You apply to Your Copyright
|
||||
and Similar Rights in Your contributions to Adapted Material in
|
||||
accordance with the terms and conditions of this Public License.
|
||||
|
||||
c. BY-SA Compatible License means a license listed at
|
||||
creativecommons.org/compatiblelicenses, approved by Creative
|
||||
Commons as essentially the equivalent of this Public License.
|
||||
|
||||
d. Copyright and Similar Rights means copyright and/or similar rights
|
||||
closely related to copyright including, without limitation,
|
||||
performance, broadcast, sound recording, and Sui Generis Database
|
||||
Rights, without regard to how the rights are labeled or
|
||||
categorized. For purposes of this Public License, the rights
|
||||
specified in Section 2(b)(1)-(2) are not Copyright and Similar
|
||||
Rights.
|
||||
|
||||
e. Effective Technological Measures means those measures that, in the
|
||||
absence of proper authority, may not be circumvented under laws
|
||||
fulfilling obligations under Article 11 of the WIPO Copyright
|
||||
Treaty adopted on December 20, 1996, and/or similar international
|
||||
agreements.
|
||||
|
||||
f. Exceptions and Limitations means fair use, fair dealing, and/or
|
||||
any other exception or limitation to Copyright and Similar Rights
|
||||
that applies to Your use of the Licensed Material.
|
||||
|
||||
g. License Elements means the license attributes listed in the name
|
||||
of a Creative Commons Public License. The License Elements of this
|
||||
Public License are Attribution and ShareAlike.
|
||||
|
||||
h. Licensed Material means the artistic or literary work, database,
|
||||
or other material to which the Licensor applied this Public
|
||||
License.
|
||||
|
||||
i. Licensed Rights means the rights granted to You subject to the
|
||||
terms and conditions of this Public License, which are limited to
|
||||
all Copyright and Similar Rights that apply to Your use of the
|
||||
Licensed Material and that the Licensor has authority to license.
|
||||
|
||||
j. Licensor means the individual(s) or entity(ies) granting rights
|
||||
under this Public License.
|
||||
|
||||
k. Share means to provide material to the public by any means or
|
||||
process that requires permission under the Licensed Rights, such
|
||||
as reproduction, public display, public performance, distribution,
|
||||
dissemination, communication, or importation, and to make material
|
||||
available to the public including in ways that members of the
|
||||
public may access the material from a place and at a time
|
||||
individually chosen by them.
|
||||
|
||||
l. Sui Generis Database Rights means rights other than copyright
|
||||
resulting from Directive 96/9/EC of the European Parliament and of
|
||||
the Council of 11 March 1996 on the legal protection of databases,
|
||||
as amended and/or succeeded, as well as other essentially
|
||||
equivalent rights anywhere in the world.
|
||||
|
||||
m. You means the individual or entity exercising the Licensed Rights
|
||||
under this Public License. Your has a corresponding meaning.
|
||||
|
||||
|
||||
Section 2 -- Scope.
|
||||
|
||||
a. License grant.
|
||||
|
||||
1. Subject to the terms and conditions of this Public License,
|
||||
the Licensor hereby grants You a worldwide, royalty-free,
|
||||
non-sublicensable, non-exclusive, irrevocable license to
|
||||
exercise the Licensed Rights in the Licensed Material to:
|
||||
|
||||
a. reproduce and Share the Licensed Material, in whole or
|
||||
in part; and
|
||||
|
||||
b. produce, reproduce, and Share Adapted Material.
|
||||
|
||||
2. Exceptions and Limitations. For the avoidance of doubt, where
|
||||
Exceptions and Limitations apply to Your use, this Public
|
||||
License does not apply, and You do not need to comply with
|
||||
its terms and conditions.
|
||||
|
||||
3. Term. The term of this Public License is specified in Section
|
||||
6(a).
|
||||
|
||||
4. Media and formats; technical modifications allowed. The
|
||||
Licensor authorizes You to exercise the Licensed Rights in
|
||||
all media and formats whether now known or hereafter created,
|
||||
and to make technical modifications necessary to do so. The
|
||||
Licensor waives and/or agrees not to assert any right or
|
||||
authority to forbid You from making technical modifications
|
||||
necessary to exercise the Licensed Rights, including
|
||||
technical modifications necessary to circumvent Effective
|
||||
Technological Measures. For purposes of this Public License,
|
||||
simply making modifications authorized by this Section 2(a)
|
||||
(4) never produces Adapted Material.
|
||||
|
||||
5. Downstream recipients.
|
||||
|
||||
a. Offer from the Licensor -- Licensed Material. Every
|
||||
recipient of the Licensed Material automatically
|
||||
receives an offer from the Licensor to exercise the
|
||||
Licensed Rights under the terms and conditions of this
|
||||
Public License.
|
||||
|
||||
b. Additional offer from the Licensor -- Adapted Material.
|
||||
Every recipient of Adapted Material from You
|
||||
automatically receives an offer from the Licensor to
|
||||
exercise the Licensed Rights in the Adapted Material
|
||||
under the conditions of the Adapter's License You apply.
|
||||
|
||||
c. No downstream restrictions. You may not offer or impose
|
||||
any additional or different terms or conditions on, or
|
||||
apply any Effective Technological Measures to, the
|
||||
Licensed Material if doing so restricts exercise of the
|
||||
Licensed Rights by any recipient of the Licensed
|
||||
Material.
|
||||
|
||||
6. No endorsement. Nothing in this Public License constitutes or
|
||||
may be construed as permission to assert or imply that You
|
||||
are, or that Your use of the Licensed Material is, connected
|
||||
with, or sponsored, endorsed, or granted official status by,
|
||||
the Licensor or others designated to receive attribution as
|
||||
provided in Section 3(a)(1)(A)(i).
|
||||
|
||||
b. Other rights.
|
||||
|
||||
1. Moral rights, such as the right of integrity, are not
|
||||
licensed under this Public License, nor are publicity,
|
||||
privacy, and/or other similar personality rights; however, to
|
||||
the extent possible, the Licensor waives and/or agrees not to
|
||||
assert any such rights held by the Licensor to the limited
|
||||
extent necessary to allow You to exercise the Licensed
|
||||
Rights, but not otherwise.
|
||||
|
||||
2. Patent and trademark rights are not licensed under this
|
||||
Public License.
|
||||
|
||||
3. To the extent possible, the Licensor waives any right to
|
||||
collect royalties from You for the exercise of the Licensed
|
||||
Rights, whether directly or through a collecting society
|
||||
under any voluntary or waivable statutory or compulsory
|
||||
licensing scheme. In all other cases the Licensor expressly
|
||||
reserves any right to collect such royalties.
|
||||
|
||||
|
||||
Section 3 -- License Conditions.
|
||||
|
||||
Your exercise of the Licensed Rights is expressly made subject to the
|
||||
following conditions.
|
||||
|
||||
a. Attribution.
|
||||
|
||||
1. If You Share the Licensed Material (including in modified
|
||||
form), You must:
|
||||
|
||||
a. retain the following if it is supplied by the Licensor
|
||||
with the Licensed Material:
|
||||
|
||||
i. identification of the creator(s) of the Licensed
|
||||
Material and any others designated to receive
|
||||
attribution, in any reasonable manner requested by
|
||||
the Licensor (including by pseudonym if
|
||||
designated);
|
||||
|
||||
ii. a copyright notice;
|
||||
|
||||
iii. a notice that refers to this Public License;
|
||||
|
||||
iv. a notice that refers to the disclaimer of
|
||||
warranties;
|
||||
|
||||
v. a URI or hyperlink to the Licensed Material to the
|
||||
extent reasonably practicable;
|
||||
|
||||
b. indicate if You modified the Licensed Material and
|
||||
retain an indication of any previous modifications; and
|
||||
|
||||
c. indicate the Licensed Material is licensed under this
|
||||
Public License, and include the text of, or the URI or
|
||||
hyperlink to, this Public License.
|
||||
|
||||
2. You may satisfy the conditions in Section 3(a)(1) in any
|
||||
reasonable manner based on the medium, means, and context in
|
||||
which You Share the Licensed Material. For example, it may be
|
||||
reasonable to satisfy the conditions by providing a URI or
|
||||
hyperlink to a resource that includes the required
|
||||
information.
|
||||
|
||||
3. If requested by the Licensor, You must remove any of the
|
||||
information required by Section 3(a)(1)(A) to the extent
|
||||
reasonably practicable.
|
||||
|
||||
b. ShareAlike.
|
||||
|
||||
In addition to the conditions in Section 3(a), if You Share
|
||||
Adapted Material You produce, the following conditions also apply.
|
||||
|
||||
1. The Adapter's License You apply must be a Creative Commons
|
||||
license with the same License Elements, this version or
|
||||
later, or a BY-SA Compatible License.
|
||||
|
||||
2. You must include the text of, or the URI or hyperlink to, the
|
||||
Adapter's License You apply. You may satisfy this condition
|
||||
in any reasonable manner based on the medium, means, and
|
||||
context in which You Share Adapted Material.
|
||||
|
||||
3. You may not offer or impose any additional or different terms
|
||||
or conditions on, or apply any Effective Technological
|
||||
Measures to, Adapted Material that restrict exercise of the
|
||||
rights granted under the Adapter's License You apply.
|
||||
|
||||
|
||||
Section 4 -- Sui Generis Database Rights.
|
||||
|
||||
Where the Licensed Rights include Sui Generis Database Rights that
|
||||
apply to Your use of the Licensed Material:
|
||||
|
||||
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
|
||||
to extract, reuse, reproduce, and Share all or a substantial
|
||||
portion of the contents of the database;
|
||||
|
||||
b. if You include all or a substantial portion of the database
|
||||
contents in a database in which You have Sui Generis Database
|
||||
Rights, then the database in which You have Sui Generis Database
|
||||
Rights (but not its individual contents) is Adapted Material,
|
||||
|
||||
including for purposes of Section 3(b); and
|
||||
c. You must comply with the conditions in Section 3(a) if You Share
|
||||
all or a substantial portion of the contents of the database.
|
||||
|
||||
For the avoidance of doubt, this Section 4 supplements and does not
|
||||
replace Your obligations under this Public License where the Licensed
|
||||
Rights include other Copyright and Similar Rights.
|
||||
|
||||
|
||||
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
|
||||
|
||||
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
|
||||
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
|
||||
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
|
||||
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
|
||||
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
|
||||
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
|
||||
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
|
||||
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
|
||||
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
|
||||
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
|
||||
|
||||
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
|
||||
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
|
||||
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
|
||||
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
|
||||
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
|
||||
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
|
||||
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
|
||||
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
|
||||
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
|
||||
|
||||
c. The disclaimer of warranties and limitation of liability provided
|
||||
above shall be interpreted in a manner that, to the extent
|
||||
possible, most closely approximates an absolute disclaimer and
|
||||
waiver of all liability.
|
||||
|
||||
|
||||
Section 6 -- Term and Termination.
|
||||
|
||||
a. This Public License applies for the term of the Copyright and
|
||||
Similar Rights licensed here. However, if You fail to comply with
|
||||
this Public License, then Your rights under this Public License
|
||||
terminate automatically.
|
||||
|
||||
b. Where Your right to use the Licensed Material has terminated under
|
||||
Section 6(a), it reinstates:
|
||||
|
||||
1. automatically as of the date the violation is cured, provided
|
||||
it is cured within 30 days of Your discovery of the
|
||||
violation; or
|
||||
|
||||
2. upon express reinstatement by the Licensor.
|
||||
|
||||
For the avoidance of doubt, this Section 6(b) does not affect any
|
||||
right the Licensor may have to seek remedies for Your violations
|
||||
of this Public License.
|
||||
|
||||
c. For the avoidance of doubt, the Licensor may also offer the
|
||||
Licensed Material under separate terms or conditions or stop
|
||||
distributing the Licensed Material at any time; however, doing so
|
||||
will not terminate this Public License.
|
||||
|
||||
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
|
||||
License.
|
||||
|
||||
|
||||
Section 7 -- Other Terms and Conditions.
|
||||
|
||||
a. The Licensor shall not be bound by any additional or different
|
||||
terms or conditions communicated by You unless expressly agreed.
|
||||
|
||||
b. Any arrangements, understandings, or agreements regarding the
|
||||
Licensed Material not stated herein are separate from and
|
||||
independent of the terms and conditions of this Public License.
|
||||
|
||||
|
||||
Section 8 -- Interpretation.
|
||||
|
||||
a. For the avoidance of doubt, this Public License does not, and
|
||||
shall not be interpreted to, reduce, limit, restrict, or impose
|
||||
conditions on any use of the Licensed Material that could lawfully
|
||||
be made without permission under this Public License.
|
||||
|
||||
b. To the extent possible, if any provision of this Public License is
|
||||
deemed unenforceable, it shall be automatically reformed to the
|
||||
minimum extent necessary to make it enforceable. If the provision
|
||||
cannot be reformed, it shall be severed from this Public License
|
||||
without affecting the enforceability of the remaining terms and
|
||||
conditions.
|
||||
|
||||
c. No term or condition of this Public License will be waived and no
|
||||
failure to comply consented to unless expressly agreed to by the
|
||||
Licensor.
|
||||
|
||||
d. Nothing in this Public License constitutes or may be interpreted
|
||||
as a limitation upon, or waiver of, any privileges and immunities
|
||||
that apply to the Licensor or You, including from the legal
|
||||
processes of any jurisdiction or authority.
|
||||
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons is not a party to its public
|
||||
licenses. Notwithstanding, Creative Commons may elect to apply one of
|
||||
its public licenses to material it publishes and in those instances
|
||||
will be considered the “Licensor.” The text of the Creative Commons
|
||||
public licenses is dedicated to the public domain under the CC0 Public
|
||||
Domain Dedication. Except for the limited purpose of indicating that
|
||||
material is shared under a Creative Commons public license or as
|
||||
otherwise permitted by the Creative Commons policies published at
|
||||
creativecommons.org/policies, Creative Commons does not authorize the
|
||||
use of the trademark "Creative Commons" or any other trademark or logo
|
||||
of Creative Commons without its prior written consent including,
|
||||
without limitation, in connection with any unauthorized modifications
|
||||
to any of its public licenses or any other arrangements,
|
||||
understandings, or agreements concerning use of licensed material. For
|
||||
the avoidance of doubt, this paragraph does not form part of the
|
||||
public licenses.
|
||||
|
||||
Creative Commons may be contacted at creativecommons.org.
|
||||
|
||||
153
nextflow/docs/Makefile
Normal file
153
nextflow/docs/Makefile
Normal file
@@ -0,0 +1,153 @@
|
||||
# Makefile for Sphinx documentation
|
||||
#
|
||||
|
||||
# You can set these variables from the command line.
|
||||
SPHINXOPTS =
|
||||
SPHINXBUILD = sphinx-build
|
||||
PAPER =
|
||||
BUILDDIR = _build
|
||||
|
||||
# Internal variables.
|
||||
PAPEROPT_a4 = -D latex_paper_size=a4
|
||||
PAPEROPT_letter = -D latex_paper_size=letter
|
||||
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
|
||||
# the i18n builder cannot share the environment and doctrees with the others
|
||||
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
|
||||
|
||||
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext
|
||||
|
||||
help:
|
||||
@echo "Please use \`make <target>' where <target> is one of"
|
||||
@echo " html to make standalone HTML files"
|
||||
@echo " dirhtml to make HTML files named index.html in directories"
|
||||
@echo " singlehtml to make a single large HTML file"
|
||||
@echo " pickle to make pickle files"
|
||||
@echo " json to make JSON files"
|
||||
@echo " htmlhelp to make HTML files and a HTML help project"
|
||||
@echo " qthelp to make HTML files and a qthelp project"
|
||||
@echo " devhelp to make HTML files and a Devhelp project"
|
||||
@echo " epub to make an epub"
|
||||
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
|
||||
@echo " latexpdf to make LaTeX files and run them through pdflatex"
|
||||
@echo " text to make text files"
|
||||
@echo " man to make manual pages"
|
||||
@echo " texinfo to make Texinfo files"
|
||||
@echo " info to make Texinfo files and run them through makeinfo"
|
||||
@echo " gettext to make PO message catalogs"
|
||||
@echo " changes to make an overview of all changed/added/deprecated items"
|
||||
@echo " linkcheck to check all external links for integrity"
|
||||
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
|
||||
|
||||
clean:
|
||||
-rm -rf $(BUILDDIR)/*
|
||||
|
||||
html:
|
||||
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
|
||||
@echo
|
||||
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
|
||||
|
||||
dirhtml:
|
||||
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
|
||||
@echo
|
||||
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
|
||||
|
||||
singlehtml:
|
||||
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
|
||||
@echo
|
||||
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
|
||||
|
||||
pickle:
|
||||
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
|
||||
@echo
|
||||
@echo "Build finished; now you can process the pickle files."
|
||||
|
||||
json:
|
||||
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
|
||||
@echo
|
||||
@echo "Build finished; now you can process the JSON files."
|
||||
|
||||
htmlhelp:
|
||||
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
|
||||
@echo
|
||||
@echo "Build finished; now you can run HTML Help Workshop with the" \
|
||||
".hhp project file in $(BUILDDIR)/htmlhelp."
|
||||
|
||||
qthelp:
|
||||
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
|
||||
@echo
|
||||
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
|
||||
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
|
||||
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Blow.qhcp"
|
||||
@echo "To view the help file:"
|
||||
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Blow.qhc"
|
||||
|
||||
devhelp:
|
||||
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
|
||||
@echo
|
||||
@echo "Build finished."
|
||||
@echo "To view the help file:"
|
||||
@echo "# mkdir -p $$HOME/.local/share/devhelp/Blow"
|
||||
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Blow"
|
||||
@echo "# devhelp"
|
||||
|
||||
epub:
|
||||
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
|
||||
@echo
|
||||
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
|
||||
|
||||
latex:
|
||||
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
|
||||
@echo
|
||||
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
|
||||
@echo "Run \`make' in that directory to run these through (pdf)latex" \
|
||||
"(use \`make latexpdf' here to do that automatically)."
|
||||
|
||||
latexpdf:
|
||||
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
|
||||
@echo "Running LaTeX files through pdflatex..."
|
||||
$(MAKE) -C $(BUILDDIR)/latex all-pdf
|
||||
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
|
||||
|
||||
text:
|
||||
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
|
||||
@echo
|
||||
@echo "Build finished. The text files are in $(BUILDDIR)/text."
|
||||
|
||||
man:
|
||||
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
|
||||
@echo
|
||||
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
|
||||
|
||||
texinfo:
|
||||
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
|
||||
@echo
|
||||
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
|
||||
@echo "Run \`make' in that directory to run these through makeinfo" \
|
||||
"(use \`make info' here to do that automatically)."
|
||||
|
||||
info:
|
||||
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
|
||||
@echo "Running Texinfo files through makeinfo..."
|
||||
make -C $(BUILDDIR)/texinfo info
|
||||
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
|
||||
|
||||
gettext:
|
||||
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
|
||||
@echo
|
||||
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
|
||||
|
||||
changes:
|
||||
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
|
||||
@echo
|
||||
@echo "The overview file is in $(BUILDDIR)/changes."
|
||||
|
||||
linkcheck:
|
||||
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
|
||||
@echo
|
||||
@echo "Link check complete; look for any errors in the above output " \
|
||||
"or in $(BUILDDIR)/linkcheck/output.txt."
|
||||
|
||||
doctest:
|
||||
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
|
||||
@echo "Testing of doctests in the sources finished, look at the " \
|
||||
"results in $(BUILDDIR)/doctest/output.txt."
|
||||
47
nextflow/docs/README.md
Normal file
47
nextflow/docs/README.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# Nextflow Documentation
|
||||
|
||||
Nextflow documentation is written using [Sphinx](http://www.sphinx-doc.org/), [MyST](https://myst-parser.readthedocs.io/en/latest/) which is an extended version of Markdown for Sphinx, and the [Read The Docs theme for Sphinx](https://github.com/readthedocs/sphinx_rtd_theme).
|
||||
|
||||
|
||||
## Dependencies
|
||||
|
||||
The most convenient approach is to create a Conda environment with Python 3.7 (other versions may work but haven't been tested).
|
||||
|
||||
The build dependencies can be installed with `pip`:
|
||||
|
||||
```bash
|
||||
cd docs
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Alternatively, you can use the Dockerfile to build the docs in a container (see below).
|
||||
|
||||
|
||||
## Contributing
|
||||
|
||||
To edit and contribute to the documentation, you only need a text editor to change the appropriate `.md` files in this directory.
|
||||
|
||||
Once you have made your changes, run the following command to build the HTML files:
|
||||
|
||||
```bash
|
||||
make clean html
|
||||
```
|
||||
|
||||
Alternatively, you can use the Dockerfile to build the docs in a container:
|
||||
|
||||
```bash
|
||||
docker build -t nextflow/sphinx:5.3.0 .
|
||||
docker run -v $(pwd):/tmp nextflow/sphinx:5.3.0 -- make html
|
||||
```
|
||||
|
||||
Then start up a local http server and open `localhost:8080` in your browser to verify the changes:
|
||||
|
||||
```bash
|
||||
python -m http.server 8080 --directory _build/html/
|
||||
```
|
||||
|
||||
|
||||
## License
|
||||
|
||||
Nextflow documentation is distributed under
|
||||
[Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license](https://creativecommons.org/licenses/by-sa/4.0/).
|
||||
BIN
nextflow/docs/_static/.DS_Store
vendored
Normal file
BIN
nextflow/docs/_static/.DS_Store
vendored
Normal file
Binary file not shown.
39
nextflow/docs/_static/dag.mmd
vendored
Normal file
39
nextflow/docs/_static/dag.mmd
vendored
Normal file
@@ -0,0 +1,39 @@
|
||||
%%{
|
||||
init: {
|
||||
'theme': 'base',
|
||||
'themeVariables': {
|
||||
'primaryColor': '#B6ECE2',
|
||||
'primaryTextColor': '#160F26',
|
||||
'primaryBorderColor': '#065647',
|
||||
'lineColor': '#545555',
|
||||
'clusterBkg': '#BABCBD22',
|
||||
'clusterBorder': '#DDDEDE',
|
||||
'fontFamily': 'arial'
|
||||
}
|
||||
}
|
||||
}%%
|
||||
flowchart TB
|
||||
subgraph " "
|
||||
v0["Channel.fromFilePairs"]
|
||||
v1["transcriptome"]
|
||||
v7["config"]
|
||||
end
|
||||
subgraph RNASEQ
|
||||
v2([INDEX])
|
||||
v3([FASTQC])
|
||||
v4([QUANT])
|
||||
end
|
||||
v8([MULTIQC])
|
||||
subgraph " "
|
||||
v9[" "]
|
||||
end
|
||||
v5(( ))
|
||||
v0 --> v3
|
||||
v0 --> v4
|
||||
v1 --> v2
|
||||
v2 --> v4
|
||||
v3 --> v5
|
||||
v4 --> v5
|
||||
v7 --> v8
|
||||
v5 --> v8
|
||||
v8 --> v9
|
||||
BIN
nextflow/docs/_static/degular/Degular-Bold.woff
vendored
Normal file
BIN
nextflow/docs/_static/degular/Degular-Bold.woff
vendored
Normal file
Binary file not shown.
BIN
nextflow/docs/_static/degular/Degular-Bold.woff2
vendored
Normal file
BIN
nextflow/docs/_static/degular/Degular-Bold.woff2
vendored
Normal file
Binary file not shown.
BIN
nextflow/docs/_static/degular/Degular-BoldItalic.woff
vendored
Normal file
BIN
nextflow/docs/_static/degular/Degular-BoldItalic.woff
vendored
Normal file
Binary file not shown.
BIN
nextflow/docs/_static/degular/Degular-BoldItalic.woff2
vendored
Normal file
BIN
nextflow/docs/_static/degular/Degular-BoldItalic.woff2
vendored
Normal file
Binary file not shown.
BIN
nextflow/docs/_static/degular/Degular-Italic.woff
vendored
Normal file
BIN
nextflow/docs/_static/degular/Degular-Italic.woff
vendored
Normal file
Binary file not shown.
BIN
nextflow/docs/_static/degular/Degular-Italic.woff2
vendored
Normal file
BIN
nextflow/docs/_static/degular/Degular-Italic.woff2
vendored
Normal file
Binary file not shown.
BIN
nextflow/docs/_static/degular/Degular-Regular.woff
vendored
Normal file
BIN
nextflow/docs/_static/degular/Degular-Regular.woff
vendored
Normal file
Binary file not shown.
BIN
nextflow/docs/_static/degular/Degular-Regular.woff2
vendored
Normal file
BIN
nextflow/docs/_static/degular/Degular-Regular.woff2
vendored
Normal file
Binary file not shown.
BIN
nextflow/docs/_static/favicon.ico
vendored
Normal file
BIN
nextflow/docs/_static/favicon.ico
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 9.4 KiB |
BIN
nextflow/docs/_static/nextflow-k8s-min.png
vendored
Normal file
BIN
nextflow/docs/_static/nextflow-k8s-min.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
BIN
nextflow/docs/_static/nextflow-logo-bg-dark.png
vendored
Normal file
BIN
nextflow/docs/_static/nextflow-logo-bg-dark.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 21 KiB |
BIN
nextflow/docs/_static/nextflow-logo-bg-light.png
vendored
Normal file
BIN
nextflow/docs/_static/nextflow-logo-bg-light.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
BIN
nextflow/docs/_static/nextflow-logo.png
vendored
Normal file
BIN
nextflow/docs/_static/nextflow-logo.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
BIN
nextflow/docs/_static/report-resource-cpu-noheader.png
vendored
Normal file
BIN
nextflow/docs/_static/report-resource-cpu-noheader.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 172 KiB |
BIN
nextflow/docs/_static/report-resource-cpu.png
vendored
Normal file
BIN
nextflow/docs/_static/report-resource-cpu.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 200 KiB |
BIN
nextflow/docs/_static/report-resource-io-read.png
vendored
Normal file
BIN
nextflow/docs/_static/report-resource-io-read.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
BIN
nextflow/docs/_static/report-resource-io-write.png
vendored
Normal file
BIN
nextflow/docs/_static/report-resource-io-write.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
BIN
nextflow/docs/_static/report-resource-job-duration.png
vendored
Normal file
BIN
nextflow/docs/_static/report-resource-job-duration.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 29 KiB |
BIN
nextflow/docs/_static/report-resource-memory-pctram.png
vendored
Normal file
BIN
nextflow/docs/_static/report-resource-memory-pctram.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
BIN
nextflow/docs/_static/report-resource-memory-ram.png
vendored
Normal file
BIN
nextflow/docs/_static/report-resource-memory-ram.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
BIN
nextflow/docs/_static/report-resource-memory-vmem.png
vendored
Normal file
BIN
nextflow/docs/_static/report-resource-memory-vmem.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
BIN
nextflow/docs/_static/report-resources-min.png
vendored
Normal file
BIN
nextflow/docs/_static/report-resources-min.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 67 KiB |
BIN
nextflow/docs/_static/report-summary-min.png
vendored
Normal file
BIN
nextflow/docs/_static/report-summary-min.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 133 KiB |
BIN
nextflow/docs/_static/report-tasks-min.png
vendored
Normal file
BIN
nextflow/docs/_static/report-tasks-min.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 104 KiB |
16
nextflow/docs/_static/seqera-logo-dark.svg
vendored
Normal file
16
nextflow/docs/_static/seqera-logo-dark.svg
vendored
Normal file
@@ -0,0 +1,16 @@
|
||||
<svg width="1479" height="500" viewBox="0 0 1479 500" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path d="M337.199 256.961L337.549 257.099L337.899 257.238L338.229 257.396L338.559 257.575L338.88 257.753L339.2 257.951L339.5 258.178L339.79 258.396L340.06 258.643L340.34 258.891L340.6 259.168L340.84 259.435L341.08 259.732L341.29 260.029L341.5 260.345L341.68 260.652L341.85 260.989L342.01 261.315L342.15 261.652L342.29 261.998L342.39 262.354L342.49 262.701L342.56 263.077L342.63 263.433L342.67 263.809L342.69 264.165L342.7 264.531L342.69 264.908L342.67 265.274L342.63 265.64L342.56 266.006L342.49 266.362L342.39 266.728L342.29 267.075L342.15 267.431L342.01 267.768L341.85 268.094L341.68 268.421L341.5 268.737L341.29 269.054L341.08 269.351L340.84 269.628L340.6 269.915L340.34 270.172L340.06 270.439L339.79 270.687L339.5 270.905L339.2 271.112L338.88 271.33L338.559 271.508L338.229 271.686L337.899 271.835L337.549 271.983L337.199 272.102L336.849 272.221L336.489 272.3L336.129 272.389L335.749 272.438L335.389 272.478L335.009 272.518H334.949L335.329 272.508L336.009 272.478L336.689 272.438L337.369 272.379L338.039 272.29L338.719 272.191L339.39 272.082L340.06 271.944L340.72 271.795L341.39 271.627L342.05 271.439L342.7 271.231L343.341 271.013L343.981 270.776L344.611 270.519L345.241 270.251L345.861 269.964L346.471 269.658L347.072 269.341L347.662 269.005L348.242 268.648L348.822 268.292L349.382 267.916L349.942 267.51L350.482 267.095L351.013 266.679L351.533 266.234L352.043 265.778L352.533 265.323L353.023 264.838L353.493 264.353L353.943 263.849L354.383 263.334L354.813 262.81L355.234 262.265L355.634 261.721L356.014 261.167L356.394 260.593L356.744 260.019L357.084 259.425L357.414 258.831L357.714 258.238L358.004 257.624L358.274 257.001L358.534 256.377L358.774 255.744L359.005 255.1L359.205 254.457L359.395 253.804L359.565 253.151L359.715 252.498L359.855 251.825L359.975 251.162L360.075 250.499L360.145 249.826L360.215 249.153L360.265 248.47L360.285 247.797L360.295 247.124L360.285 246.451L360.265 245.768L360.215 245.096L360.145 244.423L360.075 243.76L359.975 243.087L359.855 242.414L359.715 241.761L359.565 241.097L359.435 240.593H335.119L334.638 240.603H277.124L336.779 256.842L337.199 256.961Z" fill="#201637"/>
|
||||
<path d="M374.918 262.849L374.868 261.998L374.808 261.157L374.728 260.306L374.628 259.465L374.518 258.633L374.378 257.792L374.217 256.961L374.057 256.13L373.867 255.308L373.657 254.487L373.437 253.666L373.187 252.854L372.927 252.052L372.657 251.241L372.357 250.449L372.047 249.668L371.717 248.876L371.377 248.104L371.017 247.332L370.647 246.57L370.246 245.828L369.836 245.076L369.406 244.343L368.966 243.611L368.516 242.899L368.036 242.186L367.556 241.493L367.056 240.81L366.886 240.593H359.434L359.564 241.097L359.714 241.761L359.854 242.414L359.974 243.087L360.074 243.76L360.144 244.423L360.214 245.096L360.264 245.768L360.284 246.451L360.294 247.124L360.284 247.797L360.264 248.47L360.214 249.153L360.144 249.826L360.074 250.499L359.974 251.162L359.854 251.825L359.714 252.498L359.564 253.151L359.394 253.804L359.204 254.457L359.004 255.1L358.774 255.744L358.534 256.377L358.274 257.001L358.003 257.624L357.713 258.238L357.413 258.831L357.083 259.425L356.743 260.019L356.393 260.593L356.013 261.167L355.633 261.721L355.233 262.265L354.813 262.81L354.383 263.334L353.942 263.849L353.492 264.353L353.022 264.838L352.532 265.323L352.042 265.778L351.532 266.234L351.012 266.679L350.482 267.095L349.941 267.51L349.381 267.916L348.821 268.292L348.241 268.648L347.661 269.005L347.071 269.341L346.471 269.658L345.86 269.964L345.24 270.251L344.61 270.519L343.98 270.776L343.34 271.013L342.7 271.231L342.05 271.439L341.389 271.627L340.719 271.795L340.059 271.944L339.389 272.082L338.719 272.191L338.039 272.29L337.368 272.379L336.688 272.438L336.008 272.478L335.328 272.508L334.638 272.518H124.986V304.443H334.638L335.498 304.433L336.348 304.413L337.208 304.363L338.059 304.294L338.909 304.215L339.759 304.126L340.609 303.997L341.449 303.869L342.29 303.72L343.13 303.542L343.97 303.354L344.79 303.156L345.62 302.928L346.451 302.691L347.261 302.443L348.071 302.157L348.871 301.87L349.671 301.563L350.462 301.246L351.242 300.91L352.022 300.543L352.792 300.167L353.552 299.781L354.303 299.376L355.043 298.96L355.783 298.515L356.493 298.069L357.213 297.604L357.923 297.119L358.614 296.625L359.294 296.11L359.964 295.585L360.634 295.041L361.284 294.487L361.914 293.933L362.535 293.349L363.145 292.755L363.745 292.152L364.325 291.528L364.905 290.905L365.465 290.261L366.005 289.608L366.536 288.945L367.056 288.262L367.556 287.58L368.036 286.887L368.516 286.174L368.966 285.462L369.406 284.729L369.836 283.997L370.246 283.255L370.647 282.503L371.017 281.741L371.377 280.979L371.717 280.197L372.047 279.415L372.357 278.633L372.657 277.832L372.927 277.03L373.187 276.229L373.437 275.407L373.657 274.596L373.867 273.774L374.057 272.943L374.217 272.122L374.378 271.291L374.518 270.449L374.628 269.608L374.728 268.767L374.808 267.926L374.868 267.075L374.918 266.234L374.948 265.392L374.958 264.541L374.948 263.69L374.918 262.849Z" fill="#0DC09D"/>
|
||||
<path d="M162.746 248.164L162.396 248.036L162.046 247.897L161.716 247.739L161.376 247.561L161.066 247.373L160.746 247.175L160.446 246.957L160.155 246.73L159.875 246.482L159.605 246.235L159.345 245.968L159.105 245.69L158.865 245.394L158.655 245.097L158.445 244.79L158.265 244.483L158.095 244.147L157.925 243.81L157.795 243.484L157.655 243.137L157.555 242.771L157.455 242.425L157.375 242.059L157.315 241.702L157.275 241.326L157.255 240.97L157.245 240.594L157.255 240.228L157.275 239.852L157.315 239.495L157.375 239.129L157.455 238.763L157.555 238.407L157.655 238.061L157.795 237.704L157.925 237.368L158.095 237.031L158.265 236.715L158.445 236.388L158.655 236.081L158.865 235.784L159.105 235.497L159.345 235.22L159.605 234.953L159.875 234.696L160.155 234.448L160.446 234.231L160.746 234.013L161.066 233.805L161.376 233.617L161.716 233.449L162.046 233.301L162.396 233.152L162.746 233.023L163.096 232.915L163.456 232.826L163.816 232.746L164.196 232.697L164.557 232.657L164.927 232.618H164.997H164.617L163.936 232.647L163.256 232.697L162.576 232.756L161.906 232.835L161.226 232.934L160.546 233.053L159.885 233.182L159.215 233.34L158.555 233.508L157.895 233.696L157.245 233.904L156.605 234.122L155.954 234.349L155.334 234.607L154.704 234.884L154.084 235.171L153.474 235.478L152.864 235.794L152.284 236.121L151.703 236.477L151.123 236.843L150.553 237.219L150.003 237.625L149.453 238.031L148.933 238.456L148.413 238.892L147.902 239.347L147.412 239.812L146.922 240.287L146.452 240.782L146.002 241.277L145.552 241.791L145.132 242.326L144.702 242.86L144.312 243.404L143.921 243.969L143.551 244.533L143.201 245.116L142.861 245.7L142.531 246.304L142.231 246.898L141.941 247.511L141.671 248.135L141.411 248.758L141.171 249.392L140.941 250.035L140.741 250.678L140.551 251.331L140.381 251.974L140.231 252.638L140.09 253.301L139.97 253.973L139.87 254.637L139.8 255.309L139.73 255.973L139.68 256.655L139.66 257.328L139.65 258.001L139.66 258.684L139.68 259.357L139.73 260.03L139.8 260.703L139.87 261.376L139.97 262.049L140.09 262.712L140.231 263.375L140.381 264.028L140.511 264.543H164.827L165.307 264.533H222.821L163.166 248.293L162.746 248.164Z" fill="#201637"/>
|
||||
<path d="M125.026 242.285L125.066 243.136L125.136 243.977L125.216 244.818L125.316 245.659L125.426 246.501L125.566 247.332L125.716 248.173L125.886 249.004L126.076 249.826L126.286 250.647L126.506 251.468L126.756 252.28L127.016 253.081L127.286 253.883L127.586 254.685L127.897 255.466L128.227 256.258L128.567 257.03L128.927 257.802L129.297 258.554L129.697 259.306L130.107 260.058L130.537 260.791L130.977 261.513L131.427 262.225L131.898 262.938L132.388 263.641L132.888 264.323L133.058 264.541H140.51L140.38 264.027L140.23 263.373L140.09 262.71L139.97 262.047L139.87 261.374L139.8 260.701L139.729 260.029L139.679 259.356L139.659 258.683L139.649 258L139.659 257.327L139.679 256.654L139.729 255.971L139.8 255.308L139.87 254.635L139.97 253.972L140.09 253.299L140.23 252.636L140.38 251.973L140.55 251.33L140.74 250.677L140.94 250.033L141.17 249.39L141.41 248.757L141.67 248.133L141.94 247.51L142.23 246.896L142.53 246.303L142.86 245.699L143.2 245.115L143.55 244.531L143.921 243.967L144.311 243.403L144.701 242.859L145.131 242.324L145.551 241.79L146.001 241.275L146.451 240.781L146.921 240.286L147.411 239.811L147.902 239.346L148.412 238.89L148.932 238.455L149.452 238.029L150.002 237.624L150.552 237.218L151.122 236.842L151.702 236.476L152.283 236.119L152.863 235.793L153.473 235.476L154.083 235.169L154.703 234.882L155.333 234.605L155.953 234.348L156.604 234.12L157.244 233.903L157.894 233.695L158.554 233.507L159.214 233.339L159.884 233.18L160.545 233.052L161.225 232.933L161.905 232.834L162.575 232.755L163.255 232.695L163.935 232.646L164.616 232.616L165.306 232.606H374.958V200.691H165.306L164.446 200.701L163.595 200.721L162.735 200.771L161.885 200.83L161.025 200.909L160.185 201.008L159.334 201.127L158.494 201.255L157.654 201.414L156.804 201.582L155.974 201.77L155.143 201.978L154.323 202.196L153.493 202.443L152.683 202.69L151.872 202.968L151.072 203.254L150.272 203.571L149.482 203.888L148.702 204.224L147.922 204.59L147.151 204.967L146.391 205.343L145.641 205.758L144.901 206.174L144.161 206.609L143.44 207.064L142.72 207.53L142.02 208.015L141.33 208.509L140.64 209.014L139.98 209.539L139.309 210.083L138.659 210.637L138.029 211.201L137.409 211.785L136.799 212.369L136.199 212.982L135.608 213.606L135.038 214.229L134.478 214.873L133.938 215.526L133.408 216.189L132.888 216.862L132.388 217.554L131.898 218.247L131.427 218.95L130.977 219.672L130.537 220.404L130.107 221.137L129.697 221.869L129.297 222.621L128.927 223.393L128.567 224.155L128.227 224.937L127.897 225.709L127.586 226.5L127.286 227.292L127.016 228.094L126.756 228.905L126.506 229.717L126.286 230.538L126.076 231.359L125.886 232.191L125.716 233.012L125.566 233.843L125.426 234.685L125.316 235.526L125.216 236.367L125.136 237.208L125.066 238.049L125.026 238.89L124.986 239.741V240.593V241.444L125.026 242.285Z" fill="#0DC09D"/>
|
||||
<path d="M162.745 176.346L162.395 176.208L162.045 176.059L161.705 175.911L161.385 175.733L161.054 175.555L160.744 175.357L160.444 175.129L160.144 174.911L159.864 174.664L159.594 174.417L159.344 174.139L159.104 173.872L158.864 173.575L158.644 173.278L158.444 172.962L158.254 172.645L158.084 172.319L157.914 171.992L157.784 171.655L157.654 171.299L157.544 170.953L157.454 170.606L157.374 170.23L157.314 169.874L157.264 169.498L157.254 169.132L157.244 168.756L157.254 168.39L157.264 168.034L157.314 167.667L157.374 167.301L157.454 166.945L157.544 166.579L157.654 166.223L157.784 165.876L157.914 165.54L158.084 165.213L158.254 164.877L158.444 164.57L158.644 164.253L158.864 163.956L159.104 163.679L159.344 163.392L159.594 163.115L159.864 162.868L160.144 162.62L160.444 162.403L160.744 162.195L161.054 161.977L161.385 161.799L161.705 161.621L162.045 161.462L162.395 161.324L162.745 161.205L163.095 161.086L163.455 160.987L163.825 160.918L164.185 160.859L164.555 160.819L164.935 160.79H165.055H164.615L163.935 160.819L163.255 160.869L162.575 160.928L161.905 161.017L161.225 161.116L160.544 161.225L159.884 161.364L159.214 161.512L158.554 161.68L157.894 161.868L157.244 162.076L156.603 162.294L155.953 162.531L155.333 162.779L154.703 163.056L154.083 163.343L153.473 163.65L152.862 163.966L152.282 164.293L151.692 164.649L151.122 165.015L150.552 165.391L150.002 165.797L149.452 166.203L148.921 166.628L148.411 167.074L147.901 167.519L147.401 167.984L146.921 168.459L146.451 168.954L146.001 169.459L145.551 169.973L145.131 170.498L144.7 171.042L144.31 171.586L143.92 172.14L143.55 172.704L143.2 173.288L142.86 173.872L142.53 174.476L142.23 175.07L141.94 175.683L141.67 176.307L141.41 176.93L141.17 177.563L140.939 178.207L140.739 178.85L140.549 179.503L140.379 180.156L140.219 180.809L140.089 181.472L139.969 182.145L139.869 182.808L139.789 183.481L139.729 184.144L139.679 184.827L139.659 185.51L139.649 186.183L139.659 186.856L139.679 187.539L139.729 188.212L139.789 188.885L139.869 189.548L139.969 190.221L140.089 190.894L140.219 191.547L140.379 192.21L140.549 192.863L140.739 193.506L140.939 194.159L141.17 194.793L141.41 195.426L141.67 196.049L141.94 196.673L142.23 197.286L142.53 197.89L142.73 198.236L142.75 198.227L143.59 197.801L144.43 197.395L145.281 196.999L146.141 196.633L147.001 196.257L147.881 195.921L148.751 195.604L149.642 195.278L150.532 195L151.432 194.723L152.332 194.466L153.233 194.219L154.143 193.991L155.063 193.793L155.983 193.615L156.893 193.437L157.824 193.288L158.754 193.15L159.684 193.031L160.624 192.932L161.555 192.853L162.495 192.784L163.415 192.744L164.375 192.724L165.306 192.705H222.82L163.155 176.465L162.745 176.346Z" fill="#201637"/>
|
||||
<path d="M334.637 344.364H335.327L336.008 344.325L336.688 344.285L337.368 344.226L338.048 344.137L338.728 344.038L339.398 343.929L340.069 343.79L340.729 343.642L341.399 343.474L342.049 343.285L342.699 343.078L343.349 342.86L343.99 342.622L344.61 342.365L345.24 342.098L345.86 341.811L346.47 341.504L347.08 341.188L347.67 340.851L348.251 340.495L348.821 340.129L349.391 339.753L349.951 339.357L350.491 338.941L351.021 338.525L351.541 338.08L352.042 337.625L352.542 337.17L353.022 336.685L353.492 336.2L353.952 335.695L354.392 335.181L354.822 334.656L355.242 334.112L355.632 333.568L356.023 333.013L356.393 332.439L356.743 331.865L357.083 331.272L357.413 330.678L357.723 330.074L358.013 329.471L358.273 328.847L358.543 328.214L358.773 327.59L359.003 326.947L359.213 326.304L359.393 325.651L359.573 324.997L359.723 324.344L359.853 323.671L359.974 323.008L360.074 322.345L360.154 321.662L360.224 320.999L360.264 320.317L360.294 319.644V318.971V318.298L360.264 317.615L360.224 316.942L360.154 316.269L360.074 315.606L359.974 314.933L359.853 314.26L359.723 313.607L359.573 312.944L359.393 312.291L359.213 311.648L359.003 310.994L358.773 310.351L358.543 309.728L358.273 309.104L358.013 308.481L357.723 307.867L357.413 307.264L357.213 306.897L357.193 306.907L356.353 307.333L355.512 307.739L354.662 308.134L353.802 308.511L352.932 308.867L352.062 309.213L351.191 309.53L350.301 309.847L349.401 310.143L348.511 310.411L347.62 310.668L346.7 310.915L345.8 311.133L344.88 311.341L343.96 311.529L343.039 311.697L342.119 311.846L341.189 311.984L340.259 312.103L339.328 312.202L338.388 312.281L337.448 312.34L336.508 312.39L335.577 312.419L334.637 312.429H277.123L336.778 328.669L337.198 328.798L337.548 328.926L337.898 329.065L338.228 329.223L338.558 329.401L338.878 329.589L339.198 329.787L339.498 330.005L339.789 330.233L340.059 330.48L340.339 330.727L340.599 330.995L340.839 331.272L341.079 331.568L341.289 331.865L341.499 332.172L341.679 332.479L341.849 332.815L342.009 333.152L342.149 333.478L342.289 333.825L342.389 334.191L342.489 334.537L342.559 334.903L342.629 335.26L342.669 335.636L342.689 335.992L342.699 336.368L342.689 336.734L342.669 337.11L342.629 337.467L342.559 337.833L342.489 338.199L342.389 338.555L342.289 338.902L342.149 339.258L342.009 339.594L341.849 339.931L341.679 340.247L341.499 340.574L341.289 340.881L341.079 341.178L340.839 341.465L340.599 341.742L340.339 342.009L340.059 342.266L339.789 342.514L339.498 342.731L339.198 342.949L338.878 343.157L338.558 343.345L338.228 343.513L337.898 343.662L337.548 343.81L337.198 343.939L336.848 344.047L336.488 344.137L336.128 344.216L335.748 344.265L335.387 344.305L335.007 344.344L334.637 344.354" fill="#201637"/>
|
||||
<path d="M374.918 334.684L374.878 333.843L374.818 333.002L374.728 332.151L374.638 331.31L374.518 330.478L374.378 329.637L374.227 328.806L374.057 327.975L373.867 327.143L373.657 326.332L373.437 325.511L373.197 324.689L372.937 323.898L372.657 323.086L372.357 322.294L372.057 321.513L371.727 320.721L371.377 319.949L371.027 319.177L370.647 318.415L370.246 317.673L369.846 316.921L369.416 316.188L368.966 315.456L368.516 314.744L368.046 314.031L367.556 313.338L367.056 312.656L366.536 311.973L366.015 311.32L365.465 310.657L364.905 310.023L364.335 309.39L363.745 308.776L363.155 308.163L362.535 307.579L361.914 306.995L361.284 306.431L360.634 305.877L359.974 305.332L359.934 305.303L358.994 305.867L358.163 306.332L357.303 306.797L357.193 306.856L357.413 307.262L357.723 307.866L358.013 308.479L358.274 309.103L358.544 309.726L358.774 310.35L359.004 310.993L359.214 311.646L359.394 312.289L359.574 312.943L359.724 313.606L359.854 314.259L359.974 314.932L360.074 315.605L360.154 316.268L360.224 316.941L360.264 317.614L360.294 318.296V318.969V319.642L360.264 320.315L360.224 320.998L360.154 321.661L360.074 322.344L359.974 323.007L359.854 323.67L359.724 324.343L359.574 324.996L359.394 325.649L359.214 326.302L359.004 326.946L358.774 327.589L358.544 328.212L358.274 328.846L358.013 329.469L357.723 330.073L357.413 330.676L357.083 331.27L356.743 331.864L356.393 332.438L356.023 333.012L355.633 333.566L355.243 334.11L354.823 334.655L354.393 335.179L353.952 335.694L353.492 336.198L353.022 336.683L352.542 337.168L352.042 337.623L351.542 338.079L351.022 338.524L350.492 338.94L349.951 339.355L349.391 339.751L348.821 340.127L348.251 340.493L347.671 340.85L347.081 341.186L346.471 341.503L345.86 341.81L345.24 342.097L344.61 342.364L343.99 342.621L343.35 342.859L342.7 343.076L342.05 343.284L341.399 343.472L340.729 343.64L340.069 343.789L339.399 343.927L338.729 344.036L338.049 344.135L337.368 344.224L336.688 344.284L336.008 344.323L335.328 344.363H334.638H124.986V376.288H334.638L335.498 376.278L336.348 376.258L337.208 376.208L338.059 376.139L338.919 376.07L339.769 375.971L340.619 375.842L341.459 375.714L342.3 375.565L343.14 375.387L343.97 375.199L344.8 375.001L345.62 374.783L346.451 374.536L347.261 374.289L348.071 374.002L348.881 373.715L349.681 373.408L350.472 373.091L351.252 372.755L352.032 372.389L352.792 372.012L353.562 371.627L354.303 371.221L355.053 370.805L355.783 370.37L356.503 369.905L357.223 369.449L357.923 368.964L358.614 368.46L359.304 367.955L359.974 367.431L360.634 366.886L361.284 366.332L361.914 365.778L362.535 365.194L363.155 364.6L363.745 363.997L364.335 363.373L364.905 362.75L365.465 362.106L366.015 361.453L366.536 360.79L367.056 360.117L367.556 359.425L368.046 358.732L368.516 358.019L368.966 357.307L369.416 356.575L369.846 355.842L370.246 355.1L370.647 354.348L371.027 353.586L371.377 352.824L371.727 352.042L372.057 351.26L372.357 350.479L372.657 349.677L372.937 348.875L373.197 348.064L373.437 347.252L373.657 346.441L373.867 345.61L374.057 344.788L374.227 343.967L374.378 343.126L374.518 342.294L374.638 341.453L374.728 340.612L374.818 339.771L374.878 338.92L374.918 338.079L374.958 337.238V336.386V335.535L374.918 334.684Z" fill="#0DC09D"/>
|
||||
<path d="M125.026 170.459L125.066 171.31L125.126 172.151L125.216 173.002L125.316 173.844L125.426 174.675L125.566 175.516L125.716 176.347L125.886 177.179L126.076 178L126.286 178.821L126.506 179.633L126.746 180.454L127.016 181.256L127.286 182.057L127.586 182.859L127.887 183.641L128.217 184.432L128.567 185.204L128.927 185.976L129.297 186.728L129.697 187.48L130.107 188.233L130.527 188.965L130.977 189.697L131.427 190.4L131.898 191.112L132.388 191.815L132.888 192.498L133.408 193.181L133.938 193.834L134.478 194.497L135.038 195.13L135.608 195.763L136.199 196.377L136.799 196.981L137.409 197.574L138.029 198.158L138.659 198.722L139.309 199.277L139.98 199.811L140.01 199.841L140.95 199.286L141.78 198.821L142.64 198.346L142.76 198.297L142.53 197.891L142.23 197.287L141.94 196.674L141.67 196.05L141.41 195.427L141.17 194.794L140.94 194.16L140.74 193.507L140.55 192.864L140.38 192.211L140.22 191.548L140.09 190.895L139.97 190.222L139.87 189.549L139.79 188.886L139.729 188.213L139.679 187.54L139.659 186.857L139.649 186.184L139.659 185.511L139.679 184.828L139.729 184.145L139.79 183.482L139.87 182.809L139.97 182.146L140.09 181.473L140.22 180.81L140.38 180.157L140.55 179.504L140.74 178.851L140.94 178.208L141.17 177.565L141.41 176.931L141.67 176.308L141.94 175.684L142.23 175.071L142.53 174.477L142.86 173.873L143.2 173.289L143.55 172.706L143.921 172.141L144.311 171.587L144.701 171.043L145.131 170.499L145.551 169.974L146.001 169.46L146.451 168.955L146.921 168.46L147.401 167.985L147.902 167.52L148.412 167.075L148.922 166.629L149.452 166.204L150.002 165.798L150.552 165.392L151.122 165.016L151.692 164.65L152.283 164.294L152.863 163.967L153.473 163.651L154.083 163.344L154.703 163.057L155.333 162.78L155.953 162.532L156.604 162.295L157.244 162.077L157.894 161.869L158.554 161.681L159.214 161.513L159.884 161.365L160.545 161.226L161.225 161.117L161.905 161.018L162.575 160.929L163.255 160.87L163.935 160.82L164.616 160.791H165.306H374.958V128.866H165.306H164.446L163.595 128.895L162.735 128.945L161.885 129.014L161.025 129.083L160.175 129.182L159.334 129.311L158.484 129.43L157.644 129.588L156.804 129.756L155.974 129.954L155.143 130.152L154.323 130.37L153.493 130.617L152.683 130.865L151.872 131.142L151.072 131.429L150.262 131.745L149.482 132.062L148.702 132.399L147.922 132.765L147.151 133.131L146.391 133.527L145.641 133.933L144.901 134.348L144.161 134.784L143.44 135.239L142.72 135.704L142.02 136.189L141.33 136.684L140.64 137.198L139.98 137.723L139.309 138.267L138.659 138.811L138.029 139.375L137.409 139.959L136.799 140.553L136.199 141.157L135.608 141.77L135.038 142.404L134.478 143.047L133.938 143.7L133.408 144.363L132.888 145.036L132.388 145.729L131.898 146.421L131.427 147.124L130.977 147.846L130.527 148.569L130.107 149.311L129.697 150.053L129.297 150.796L128.927 151.567L128.567 152.329L128.217 153.111L127.887 153.893L127.586 154.675L127.286 155.466L127.016 156.278L126.746 157.08L126.506 157.891L126.286 158.712L126.076 159.534L125.886 160.365L125.716 161.186L125.566 162.018L125.426 162.859L125.316 163.7L125.216 164.541L125.126 165.382L125.066 166.233L125.026 167.075L124.986 167.916V168.767V169.618L125.026 170.459Z" fill="#0DC09D"/>
|
||||
<path d="M516.784 329.897H453.084V305.262H516.784C526.645 305.262 534.243 303.388 539.576 299.619C544.909 295.86 547.581 291.007 547.581 285.071C547.581 279.134 545.058 274.78 540.033 272.007C534.998 269.244 527.549 266.959 517.687 265.182L507.418 263.405C497.348 261.628 488.191 259.05 479.948 255.692C471.695 252.333 465.15 247.675 460.324 241.739C455.497 235.803 453.074 228.187 453.074 218.882C453.074 205.037 458.308 194.297 468.775 186.681C479.243 179.065 493.127 175.257 510.437 175.257H572.627V200.194H510.437C501.986 200.194 495.342 201.727 490.515 204.793C485.689 207.859 483.265 212.165 483.265 217.701C483.265 223.637 485.629 227.992 490.356 230.765C495.084 233.537 501.479 235.617 509.524 236.994L520.091 238.771C530.757 240.548 540.519 243.028 549.378 246.191C558.237 249.355 565.229 253.905 570.363 259.841C575.498 265.778 578.06 273.696 578.06 283.587C578.06 298.232 572.518 309.607 561.455 317.721C550.381 325.835 535.494 329.887 516.774 329.887L516.784 329.897Z" fill="#201637"/>
|
||||
<path d="M892.291 377.653V307.648H887.432C885.202 311.665 881.963 315.525 877.704 319.226C873.445 322.937 868.026 325.995 861.447 328.4C854.858 330.804 846.71 332.012 836.982 332.012C824.414 332.012 812.866 329.003 802.328 322.987C791.79 316.97 783.382 308.301 777.103 296.96C770.824 285.629 767.685 271.942 767.685 255.891V251.378C767.685 235.337 770.874 221.65 777.263 210.309C783.642 198.978 792.1 190.299 802.638 184.283C813.176 178.266 824.624 175.257 836.982 175.257C851.569 175.257 862.767 177.87 870.565 183.075C878.364 188.291 884.193 194.208 888.042 200.829H892.901V175.257H923.605V377.633H892.301L892.291 377.653ZM845.79 304.936C859.567 304.936 870.815 300.621 879.523 292.002C888.232 283.382 892.591 271.042 892.591 255V252.289C892.591 236.445 888.182 224.213 879.373 215.584C870.555 206.964 859.357 202.65 845.79 202.65C832.223 202.65 821.325 206.964 812.506 215.584C803.688 224.213 799.289 236.445 799.289 252.289V255C799.289 271.052 803.698 283.382 812.506 292.002C821.325 300.631 832.413 304.936 845.79 304.936Z" fill="#201637"/>
|
||||
<path d="M744.698 249.35C744.698 234.016 741.685 220.629 735.66 209.18C729.636 197.73 721.242 188.782 710.51 182.345C699.759 175.909 687.253 172.681 673.003 172.681C658.754 172.681 645.485 175.95 634.446 182.497C628.916 185.765 623.991 189.669 619.651 194.229C619.106 194.794 618.571 195.379 618.036 195.964C616.45 197.74 614.974 199.596 613.567 201.533C611.704 204.125 609.97 206.879 608.384 209.795C602.062 221.447 598.891 235.146 598.891 250.893V254.575C598.891 270.323 602.052 284.012 608.384 295.674C614.706 307.326 623.496 316.375 634.743 322.811C645.98 329.247 659.14 332.475 674.202 332.475C687.461 332.475 698.5 330.326 707.339 326.039C716.179 321.741 723.204 316.425 728.427 310.09C729.873 308.324 731.241 306.589 732.519 304.874C735.849 300.425 738.604 296.138 740.774 291.992L714.871 278.494C711.66 285.455 707.082 291.588 701.166 296.895C695.23 302.211 686.45 304.864 674.807 304.864C662.351 304.864 651.956 300.92 643.632 293.061C643.117 292.567 642.611 292.083 642.126 291.558C641.63 291.033 641.164 290.519 640.699 289.994C639.589 288.693 638.558 287.331 637.607 285.899C633.554 279.705 631.235 272.33 630.65 263.786C630.591 262.969 630.561 262.141 630.532 261.314H744.708V249.35H744.698ZM651.312 237.083L635.447 253.234H630.224C630.145 248.583 630.224 242.611 630.829 237.083C631.126 234.985 631.523 232.957 632.038 231.01C632.533 229.063 633.138 227.207 633.841 225.431C636.18 219.51 639.658 214.496 644.236 210.4C651.559 203.853 661.043 200.585 672.706 200.585C684.369 200.585 693.793 203.853 701.027 210.4C708.261 216.947 712.373 225.845 713.374 237.083H651.322H651.312Z" fill="#201637"/>
|
||||
<path d="M1093.62 249.35C1093.62 234.016 1090.6 220.629 1084.58 209.18C1078.55 197.73 1070.16 188.782 1059.43 182.345C1048.68 175.909 1036.17 172.681 1021.92 172.681C1007.67 172.681 994.404 175.95 983.365 182.497C977.835 185.765 972.91 189.669 968.57 194.229C968.025 194.794 967.49 195.379 966.955 195.964C965.369 197.74 963.893 199.596 962.486 201.533C960.623 204.125 958.889 206.879 957.303 209.795C950.981 221.447 947.81 235.146 947.81 250.893V254.575C947.81 270.323 950.971 284.012 957.303 295.674C963.625 307.326 972.415 316.375 983.662 322.811C994.899 329.247 1008.06 332.475 1023.12 332.475C1036.38 332.475 1047.42 330.326 1056.26 326.039C1065.1 321.741 1072.12 316.425 1077.35 310.09C1078.79 308.324 1080.16 306.589 1081.44 304.874C1084.77 300.425 1087.52 296.138 1089.69 291.992L1063.79 278.494C1060.58 285.455 1056 291.588 1050.08 296.895C1044.15 302.211 1035.37 304.864 1023.73 304.864C1011.27 304.864 1000.87 300.92 992.551 293.061C992.036 292.567 991.53 292.083 991.045 291.558C990.549 291.033 990.083 290.519 989.618 289.994C988.508 288.693 987.477 287.331 986.526 285.899C982.473 279.705 980.154 272.33 979.569 263.786C979.51 262.969 979.48 262.141 979.451 261.314H1093.63V249.35H1093.62ZM1000.23 237.083L984.366 253.234H979.143C979.064 248.583 979.143 242.611 979.748 237.083C980.045 234.985 980.441 232.957 980.957 231.01C981.452 229.063 982.057 227.207 982.76 225.431C985.099 219.51 988.577 214.496 993.155 210.4C1000.48 203.853 1009.96 200.585 1021.62 200.585C1033.29 200.585 1042.71 203.853 1049.95 210.4C1057.18 216.947 1061.29 225.845 1062.29 237.083H1000.24H1000.23Z" fill="#201637"/>
|
||||
<path d="M1124.88 330.029V173.285H1155.58V190.732H1160.44C1162.87 184.517 1166.77 179.955 1172.13 177.045C1177.5 174.136 1184.14 172.681 1192.04 172.681H1209.98V200.657H1190.83C1180.69 200.657 1172.39 203.418 1165.91 208.931C1159.42 214.453 1156.18 222.924 1156.18 234.354V330.019H1124.87L1124.88 330.029Z" fill="#201637"/>
|
||||
<path d="M1354.01 230.335C1354.01 211.934 1348.37 197.721 1337.05 187.707C1325.75 177.693 1310.23 172.681 1290.48 172.681C1277.53 172.681 1266.58 174.72 1257.63 178.818C1248.69 182.916 1241.49 188.33 1236.05 195.07C1230.6 201.819 1226.64 209.182 1224.17 217.147L1253.77 226.659C1255.62 218.483 1259.43 211.834 1265.19 206.721C1270.96 201.619 1279.27 199.057 1290.17 199.057C1301.07 199.057 1309.5 201.769 1314.84 207.183C1320.2 212.597 1322.86 219.608 1322.86 228.196V254.351H1317.73L1301.28 238.009H1276.6C1259.95 238.009 1246.12 241.946 1235.12 249.821C1224.12 257.686 1218.61 269.297 1218.61 284.634C1218.61 294.859 1221.14 303.538 1226.17 310.699C1231.21 317.861 1238 323.274 1246.52 326.951C1255.07 330.627 1264.78 332.475 1275.66 332.475C1286.54 332.475 1294.58 330.988 1300.95 328.025C1307.33 325.062 1312.17 321.587 1315.45 317.599C1318.75 313.612 1321.2 309.976 1322.85 306.712H1327.79V329.898L1354 329.898V230.345L1354.01 230.335ZM1322.86 265.6C1322.86 278.286 1318.9 288.25 1310.98 295.502C1303.07 302.754 1292.85 306.39 1280.3 306.39C1271.05 306.39 1263.75 304.351 1258.4 300.253C1253.05 296.155 1250.38 290.641 1250.38 283.69C1250.38 276.739 1252.96 271.466 1258.09 267.901C1263.24 264.325 1270.01 262.537 1278.44 262.537H1322.85V265.6H1322.86Z" fill="#201637"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 28 KiB |
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user