Feature Specification: Fusion GPU Metrics Collection

Feature Branch: 260410-fusion-gpu-metrics
Created: 2026-04-10
Status: Draft
Input: User description: "Collect GPU metrics from Fusion trace.json and send to Seqera Platform via TowerClient"

User Scenarios & Testing (mandatory)

User Story 1 - GPU metrics sent to Platform on task completion (Priority: P1)

A user runs a Nextflow pipeline with Fusion enabled on a GPU-equipped executor (e.g., AWS Batch, Google Batch, Kubernetes). When each task completes, Nextflow reads the Fusion-generated .fusion/trace.json file from the task work directory, extracts the gpu block, and includes it in the task trace data sent to Seqera Platform. The user can then view GPU utilization metrics (compute %, memory %, active time, etc.) for each task in the Platform UI.

Why this priority: This is the core feature. Without it, GPU usage is invisible to Platform users running Fusion-enabled pipelines.

Independent Test: Can be tested by running a Fusion-enabled task that produces a .fusion/trace.json with a gpu block, then verifying the GPU data appears in the task payload sent to Platform.

Acceptance Scenarios:

Given a completed task with Fusion enabled and a valid .fusion/trace.json containing a gpu block, When the task trace is collected, Then all GPU metrics from the gpu block are included in the task data sent to Platform.
Given a completed task with Fusion enabled and a valid .fusion/trace.json without a gpu block (CPU-only task), When the task trace is collected, Then no GPU metrics are sent and no error occurs.
Given a failed task with Fusion enabled and a valid .fusion/trace.json containing a gpu block, When the task trace is collected, Then GPU metrics are still sent (metrics are collected irrespective of task status).

User Story 2 - Graceful handling when trace.json is missing or malformed (Priority: P2)

When Fusion's .fusion/trace.json file is missing (e.g., task was killed before Fusion wrote it) or contains invalid JSON, the system logs a debug-level warning and proceeds without GPU metrics. The task trace is still sent to Platform with all other fields intact.

Why this priority: Robustness is essential — GPU metrics are supplementary data and must never cause task reporting to fail.

Independent Test: Can be tested by simulating a completed task where .fusion/trace.json is absent or contains malformed JSON, and verifying the task trace is still sent successfully without GPU data.

Acceptance Scenarios:

Given a completed Fusion-enabled task where .fusion/trace.json does not exist, When the task trace is collected, Then no GPU metrics are included and no error is raised.
Given a completed Fusion-enabled task where .fusion/trace.json contains invalid JSON, When the task trace is collected, Then the file is skipped with a debug log message and the task trace is sent without GPU data.
Given a completed Fusion-enabled task where .fusion/trace.json exists but the gpu block is null/absent, When the task trace is collected, Then no GPU metrics are included and no error is raised.

Edge Cases

What happens when the gpu block contains unexpected or extra fields not in the known schema? They are included as-is (forward compatibility).
What happens when Fusion is not enabled for a task? No attempt is made to read .fusion/trace.json.
What happens when the task work directory is inaccessible at trace collection time (e.g., remote storage timeout)? The same error handling as existing .command.trace parsing applies — log and continue.

Requirements (mandatory)

Functional Requirements

FR-001: System MUST read the file .fusion/trace.json from the task work directory on task completion when the executor has Fusion enabled.
FR-002: System MUST extract the entire gpu block from the parsed trace.json as a map.
FR-003: System MUST store the GPU metrics as a transient field on TraceRecord (following the same pattern as resourceAllocation).
FR-004: System MUST include the GPU metrics map in the task payload sent to Seqera Platform via the Tower observer.
FR-005: System MUST collect GPU metrics irrespective of task completion status (success or failure).
FR-006: System MUST NOT fail or disrupt task trace reporting if .fusion/trace.json is missing, unreadable, or malformed.
FR-007: System MUST only attempt to read .fusion/trace.json when Fusion is enabled for the executor.

Key Entities

Fusion Trace File: JSON file at .fusion/trace.json in the task work directory, produced by the Fusion client. Contains proc, gpu, and cgroup blocks with runtime metrics.
GPU Metrics Block: The gpu object within trace.json, containing fields: name, mem, driver, active_time, pct, peak, pct_mem, peak_mem, avg_mem, peak_mem_used, avg_mem_bw_util, peak_mem_bw_util.

Example `.fusion/trace.json`

{
  "proc": {
    "realtime": 660541,
    "pct_cpu": 1045,
    "cpu_name": "Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz",
    "arch": "linux/amd64",
    "rchar": 14112539262,
    "wchar": 12668821375,
    "syscr": 1823378,
    "syscw": 169293,
    "read_bytes": 8011776,
    "write_bytes": 102400,
    "pct_mem": 56,
    "vmem": 39015152,
    "rss": 14826068,
    "peak_vmem": 39047920,
    "peak_rss": 15775480,
    "vol_ctxt": 413015,
    "inv_ctxt": 1540
  },
  "gpu": {
    "name": "Tesla T4",
    "mem": 15360,
    "driver": "580.126.09",
    "active_time": 651030,
    "pct": 75,
    "peak": 100,
    "pct_mem": 40.11115345483025,
    "peak_mem": 74.140625,
    "avg_mem": 6161,
    "peak_mem_used": 11388,
    "avg_mem_bw_util": 43,
    "peak_mem_bw_util": 83
  },
  "cgroup": {
    "version": "v2",
    "memory_current": 25469927424,
    "memory_peak": 41178980352,
    "memory_rss": 67919872,
    "memory_peak_rss": 14783070208,
    "cpu_usage_usec": 785302059,
    "cpu_user_usec": 549732867,
    "cpu_system_usec": 235569192,
    "io_read_bytes": 8503296,
    "io_write_bytes": 12671918080,
    "io_read_ops": 98,
    "io_write_ops": 97975,
    "memory_limit": 77309411328,
    "cpu_quota": 0,
    "cpu_period": 0,
    "memory_oom_kills": 0,
    "cpu_nr_throttled": 0,
    "cpu_throttled_usec": 0,
    "cpu_psi_some": 582969,
    "cpu_psi_full": 582860,
    "memory_psi_some": 0,
    "memory_psi_full": 0,
    "io_psi_some": 1038270,
    "io_psi_full": 1037514
  }
}

TraceRecord GPU field: New transient field on TraceRecord that carries the GPU metrics map through the existing trace pipeline to the Tower observer, following the resourceAllocation pattern.

Success Criteria (mandatory)

Measurable Outcomes

SC-001: GPU metrics from Fusion trace files are visible in Seqera Platform for all Fusion-enabled tasks that ran on GPU hardware.
SC-002: Tasks without GPU usage or without Fusion enabled report successfully with no GPU data and no errors.
SC-003: A missing or malformed .fusion/trace.json does not cause any task to fail reporting — 100% of tasks still have their standard metrics delivered.
SC-004: GPU metrics collection adds negligible overhead — reading and parsing a single small JSON file per task completion.

Assumptions

The Fusion client is responsible for creating .fusion/trace.json in the task work directory. Nextflow only reads it.
The gpu block schema may evolve over time. The implementation forwards the entire block as a map rather than mapping to fixed fields, ensuring forward compatibility.
Seqera Platform API already accepts or will be updated to accept the GPU metrics payload alongside existing task trace data.
The file path .fusion/trace.json is stable and defined by the Fusion client contract.
All executors that support Fusion (AWS Batch, Google Batch, Azure Batch, Kubernetes, Seqera, SLURM) benefit from this feature without executor-specific code — the detection is based on whether Fusion is enabled, not on the executor type.

8.0 KiB Raw Permalink Blame History