Compare commits
7 Commits
b0a5447797
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 437425e652 | |||
| 229c8c358d | |||
| 02cefd8fa9 | |||
| 27655dc1b8 | |||
| dfae78af71 | |||
| 4188bea061 | |||
| f227d054b8 |
10
k8s-dvfs/.gitignore
vendored
Normal file
10
k8s-dvfs/.gitignore
vendored
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
# Ignore Gradle project-specific cache directory
|
||||||
|
.gradle
|
||||||
|
.idea
|
||||||
|
.nextflow*
|
||||||
|
|
||||||
|
# Ignore Gradle build output directory
|
||||||
|
build
|
||||||
|
work
|
||||||
|
lib/groovy-6.0.0-beta-1.jar lib/groovy-6.0.0-beta-1-sources.jar lib/groovy-ant-6.0.0-beta-1.jar lib/groovy-ant-6.0.0-beta-1-sources.jar lib/groovy-astbuilder-6.0.0-beta-1.jar lib/groovy-astbuilder-6.0.0-beta-1-sources.jar lib/groovy-cli-commons-6.0.0-beta-1.jar lib/groovy-cli-commons-6.0.0-beta-1-sources.jar lib/groovy-cli-picocli-6.0.0-beta-1.jar lib/groovy-cli-picocli-6.0.0-beta-1-sources.jar lib/groovy-console-6.0.0-beta-1.jar lib/groovy-console-6.0.0-beta-1-sources.jar lib/groovy-contracts-6.0.0-beta-1.jar lib/groovy-contracts-6.0.0-beta-1-sources.jar lib/groovy-csv-6.0.0-beta-1.jar lib/groovy-csv-6.0.0-beta-1-sources.jar lib/groovy-datetime-6.0.0-beta-1.jar lib/groovy-datetime-6.0.0-beta-1-sources.jar lib/groovy-dateutil-6.0.0-beta-1.jar lib/groovy-dateutil-6.0.0-beta-1-sources.jar lib/groovy-docgenerator-6.0.0-beta-1.jar lib/groovy-docgenerator-6.0.0-beta-1-sources.jar lib/groovy-ginq-6.0.0-beta-1.jar lib/groovy-ginq-6.0.0-beta-1-sources.jar lib/groovy-grape-ivy-6.0.0-beta-1.jar lib/groovy-grape-ivy-6.0.0-beta-1-sources.jar lib/groovy-grape-maven-6.0.0-beta-1.jar lib/groovy-grape-maven-6.0.0-beta-1-sources.jar lib/groovy-groovydoc-6.0.0-beta-1.jar lib/groovy-groovydoc-6.0.0-beta-1-sources.jar lib/groovy-groovysh-6.0.0-beta-1.jar lib/groovy-groovysh-6.0.0-beta-1-sources.jar lib/groovy-http-builder-6.0.0-beta-1.jar lib/groovy-http-builder-6.0.0-beta-1-sources.jar lib/groovy-jmx-6.0.0-beta-1.jar lib/groovy-jmx-6.0.0-beta-1-sources.jar lib/groovy-json-6.0.0-beta-1.jar lib/groovy-json-6.0.0-beta-1-sources.jar lib/groovy-jsr223-6.0.0-beta-1.jar lib/groovy-jsr223-6.0.0-beta-1-sources.jar lib/groovy-macro-6.0.0-beta-1.jar lib/groovy-macro-6.0.0-beta-1-sources.jar lib/groovy-macro-library-6.0.0-beta-1.jar lib/groovy-macro-library-6.0.0-beta-1-sources.jar lib/groovy-markdown-6.0.0-beta-1.jar lib/groovy-markdown-6.0.0-beta-1-sources.jar lib/groovy-nio-6.0.0-beta-1.jar lib/groovy-nio-6.0.0-beta-1-sources.jar lib/groovy-reactor-6.0.0-beta-1.jar lib/groovy-reactor-6.0.0-beta-1-sources.jar lib/groovy-rxjava-6.0.0-beta-1.jar lib/groovy-rxjava-6.0.0-beta-1-sources.jar lib/groovy-servlet-6.0.0-beta-1.jar lib/groovy-servlet-6.0.0-beta-1-sources.jar lib/groovy-sql-6.0.0-beta-1.jar lib/groovy-sql-6.0.0-beta-1-sources.jar lib/groovy-swing-6.0.0-beta-1.jar lib/groovy-swing-6.0.0-beta-1-sources.jar lib/groovy-templates-6.0.0-beta-1.jar lib/groovy-templates-6.0.0-beta-1-sources.jar lib/groovy-test-6.0.0-beta-1.jar lib/groovy-test-6.0.0-beta-1-sources.jar lib/groovy-test-junit5-6.0.0-beta-1.jar lib/groovy-test-junit5-6.0.0-beta-1-sources.jar lib/groovy-test-junit6-6.0.0-beta-1.jar lib/groovy-test-junit6-6.0.0-beta-1-sources.jar lib/groovy-testng-6.0.0-beta-1.jar lib/groovy-testng-6.0.0-beta-1-sources.jar lib/groovy-toml-6.0.0-beta-1.jar lib/groovy-toml-6.0.0-beta-1-sources.jar lib/groovy-typecheckers-6.0.0-beta-1.jar lib/groovy-typecheckers-6.0.0-beta-1-sources.jar lib/groovy-xml-6.0.0-beta-1.jar lib/groovy-xml-6.0.0-beta-1-sources.jar lib/groovy-yaml-6.0.0-beta-1.jar lib/groovy-yaml-6.0.0-beta-1-sources.jar
|
||||||
|
lib
|
||||||
25
k8s-dvfs/Makefile
Normal file
25
k8s-dvfs/Makefile
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
# Use the Gradle wrapper by default; override with e.g. `make GRADLE=gradle ...`
|
||||||
|
# or `export GRADLE=gradle` (useful in pixi/conda environments).
|
||||||
|
GRADLE ?= ./gradlew
|
||||||
|
|
||||||
|
# Build the plugin
|
||||||
|
assemble:
|
||||||
|
$(GRADLE) assemble
|
||||||
|
|
||||||
|
clean:
|
||||||
|
rm -rf .nextflow*
|
||||||
|
rm -rf work
|
||||||
|
rm -rf build
|
||||||
|
$(GRADLE) clean
|
||||||
|
|
||||||
|
# Run plugin unit tests
|
||||||
|
test:
|
||||||
|
$(GRADLE) test
|
||||||
|
|
||||||
|
# Install the plugin into local nextflow plugins dir
|
||||||
|
install:
|
||||||
|
$(GRADLE) install
|
||||||
|
|
||||||
|
# Publish the plugin
|
||||||
|
release:
|
||||||
|
$(GRADLE) releasePlugin
|
||||||
86
k8s-dvfs/README.md
Normal file
86
k8s-dvfs/README.md
Normal file
@@ -0,0 +1,86 @@
|
|||||||
|
# k8s-dvfs
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
`k8s-dvfs` is a Nextflow plugin scaffolded from the official plugin
|
||||||
|
template. Out of the box it provides:
|
||||||
|
|
||||||
|
- A custom function `sayHello` that can be imported into Nextflow scripts.
|
||||||
|
- A workflow observer that reacts to pipeline lifecycle events (start and
|
||||||
|
completion).
|
||||||
|
|
||||||
|
Replace this section with a description of what your plugin actually does.
|
||||||
|
|
||||||
|
Note: The **Summary**, **Get Started**, **Examples**, and **License** sections are
|
||||||
|
mandatory: they are required by the Nextflow Registry, which uses this
|
||||||
|
file as the plugin description. The **Plugin development** section below is
|
||||||
|
guidance for working on the plugin and can be removed before publishing.
|
||||||
|
|
||||||
|
## Get Started
|
||||||
|
|
||||||
|
Enable the plugin in your pipeline `nextflow.config`:
|
||||||
|
|
||||||
|
```groovy
|
||||||
|
plugins {
|
||||||
|
id 'k8s-dvfs@0.1.0'
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Nextflow downloads the plugin from the Nextflow Registry the first time
|
||||||
|
the pipeline runs.
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
Import and call the `sayHello` function from a Nextflow script:
|
||||||
|
|
||||||
|
```nextflow
|
||||||
|
include { sayHello } from 'plugin/k8s-dvfs'
|
||||||
|
|
||||||
|
workflow {
|
||||||
|
channel.of('Mundo', 'World').map { target -> sayHello(target) }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The bundled observer prints a message when the pipeline starts and completes,
|
||||||
|
so running any pipeline with the plugin enabled produces:
|
||||||
|
|
||||||
|
```
|
||||||
|
Pipeline is starting! 🚀
|
||||||
|
Pipeline complete! 👋
|
||||||
|
```
|
||||||
|
|
||||||
|
## Plugin development
|
||||||
|
|
||||||
|
This project was created from the [Nextflow plugin template](https://www.nextflow.io/docs/latest/guides/gradle-plugin.html#gradle-plugin-create).
|
||||||
|
|
||||||
|
### Building
|
||||||
|
|
||||||
|
To build the plugin:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make assemble
|
||||||
|
```
|
||||||
|
|
||||||
|
### Testing with Nextflow
|
||||||
|
|
||||||
|
The plugin can be tested without a local Nextflow installation:
|
||||||
|
|
||||||
|
1. Build and install the plugin to your local Nextflow installation: `make install`
|
||||||
|
2. Run a pipeline with the plugin: `nextflow run hello -plugins k8s-dvfs@0.1.0`
|
||||||
|
|
||||||
|
### Publishing
|
||||||
|
|
||||||
|
Plugins can be published to a central Nextflow registry to make them accessible to the Nextflow community.
|
||||||
|
|
||||||
|
Follow these steps to publish the plugin to the Nextflow Registry:
|
||||||
|
|
||||||
|
1. Create a file named `$HOME/.gradle/gradle.properties`, where `$HOME` is your home directory. Add the following properties:
|
||||||
|
* `npr.apiKey`: Your Nextflow Registry access token.
|
||||||
|
2. Package your plugin and publish it to the registry: `make release`.
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Apache License 2.0. See the [`COPYING`](COPYING) file for details.
|
||||||
|
|
||||||
|
Note: The above license is given for guidance only; however the Nextflow Registry
|
||||||
|
requires the plugin to include an OSS (open source software) license.
|
||||||
51
k8s-dvfs/build.gradle
Normal file
51
k8s-dvfs/build.gradle
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
plugins {
|
||||||
|
id 'io.nextflow.nextflow-plugin' version '1.0.0-beta.15'
|
||||||
|
}
|
||||||
|
|
||||||
|
version = '0.1.0'
|
||||||
|
|
||||||
|
nextflowPlugin {
|
||||||
|
nextflowVersion = '26.04.6'
|
||||||
|
|
||||||
|
provider = 'recreational.tech'
|
||||||
|
className = 'recreationaltech.plugin.K8sPlugin'
|
||||||
|
useDefaultDependencies = true
|
||||||
|
generateSpec = false
|
||||||
|
extensionPoints = [
|
||||||
|
'recreationaltech.plugin.K8sConfig',
|
||||||
|
'recreationaltech.plugin.K8sExecutor',
|
||||||
|
'recreationaltech.plugin.cli.KubeCommandImpl',
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
sourceSets {
|
||||||
|
main.java.srcDirs = []
|
||||||
|
main.groovy.srcDirs = ['src/main/groovy']
|
||||||
|
main.resources.srcDirs = ['src/resources']
|
||||||
|
test.groovy.srcDirs = ['src/test/groovy']
|
||||||
|
test.java.srcDirs = []
|
||||||
|
test.resources.srcDirs = []
|
||||||
|
}
|
||||||
|
|
||||||
|
configurations {
|
||||||
|
// see https://docs.gradle.org/4.1/userguide/dependency_management.html#sub:exclude_transitive_dependencies
|
||||||
|
runtimeClasspath.exclude group: 'org.slf4j', module: 'slf4j-api'
|
||||||
|
}
|
||||||
|
|
||||||
|
dependencies {
|
||||||
|
//compileOnly project(':nextflow')
|
||||||
|
compileOnly 'org.slf4j:slf4j-api:2.0.17'
|
||||||
|
compileOnly 'org.pf4j:pf4j:3.14.1'
|
||||||
|
compileOnly 'dev.failsafe:failsafe:3.3.2'
|
||||||
|
|
||||||
|
implementation 'org.bouncycastle:bcprov-ext-jdk18on:1.78.1'
|
||||||
|
implementation 'org.bouncycastle:bcpkix-jdk18on:1.84'
|
||||||
|
|
||||||
|
//testImplementation(testFixtures(project(":nextflow")))
|
||||||
|
testImplementation "org.apache.groovy:groovy:4.0.31"
|
||||||
|
testImplementation "org.apache.groovy:groovy-nio:4.0.31"
|
||||||
|
}
|
||||||
|
|
||||||
|
test {
|
||||||
|
useJUnitPlatform()
|
||||||
|
}
|
||||||
BIN
k8s-dvfs/gradle/wrapper/gradle-wrapper.jar
vendored
Normal file
BIN
k8s-dvfs/gradle/wrapper/gradle-wrapper.jar
vendored
Normal file
Binary file not shown.
@@ -1,6 +1,6 @@
|
|||||||
distributionBase=GRADLE_USER_HOME
|
distributionBase=GRADLE_USER_HOME
|
||||||
distributionPath=wrapper/dists
|
distributionPath=wrapper/dists
|
||||||
distributionUrl=https\://services.gradle.org/distributions/gradle-9.3.1-bin.zip
|
distributionUrl=https\://services.gradle.org/distributions/gradle-8.14-bin.zip
|
||||||
networkTimeout=10000
|
networkTimeout=10000
|
||||||
validateDistributionUrl=true
|
validateDistributionUrl=true
|
||||||
zipStoreBase=GRADLE_USER_HOME
|
zipStoreBase=GRADLE_USER_HOME
|
||||||
12
nextflow/gradlew → k8s-dvfs/gradlew
vendored
12
nextflow/gradlew → k8s-dvfs/gradlew
vendored
@@ -15,8 +15,6 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
#
|
#
|
||||||
# SPDX-License-Identifier: Apache-2.0
|
|
||||||
#
|
|
||||||
|
|
||||||
##############################################################################
|
##############################################################################
|
||||||
#
|
#
|
||||||
@@ -57,7 +55,7 @@
|
|||||||
# Darwin, MinGW, and NonStop.
|
# Darwin, MinGW, and NonStop.
|
||||||
#
|
#
|
||||||
# (3) This script is generated from the Groovy template
|
# (3) This script is generated from the Groovy template
|
||||||
# https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
|
# https://github.com/gradle/gradle/blob/HEAD/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
|
||||||
# within the Gradle project.
|
# within the Gradle project.
|
||||||
#
|
#
|
||||||
# You can find Gradle at https://github.com/gradle/gradle/.
|
# You can find Gradle at https://github.com/gradle/gradle/.
|
||||||
@@ -86,7 +84,7 @@ done
|
|||||||
# shellcheck disable=SC2034
|
# shellcheck disable=SC2034
|
||||||
APP_BASE_NAME=${0##*/}
|
APP_BASE_NAME=${0##*/}
|
||||||
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
|
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
|
||||||
APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) || exit
|
APP_HOME=$( cd "${APP_HOME:-./}" > /dev/null && pwd -P ) || exit
|
||||||
|
|
||||||
# Use the maximum available, or set MAX_FD != -1 to use that value.
|
# Use the maximum available, or set MAX_FD != -1 to use that value.
|
||||||
MAX_FD=maximum
|
MAX_FD=maximum
|
||||||
@@ -114,7 +112,7 @@ case "$( uname )" in #(
|
|||||||
NONSTOP* ) nonstop=true ;;
|
NONSTOP* ) nonstop=true ;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
CLASSPATH="\\\"\\\""
|
CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
|
||||||
|
|
||||||
|
|
||||||
# Determine the Java command to use to start the JVM.
|
# Determine the Java command to use to start the JVM.
|
||||||
@@ -205,7 +203,7 @@ fi
|
|||||||
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
|
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
|
||||||
|
|
||||||
# Collect all arguments for the java command:
|
# Collect all arguments for the java command:
|
||||||
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
|
# * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
|
||||||
# and any embedded shellness will be escaped.
|
# and any embedded shellness will be escaped.
|
||||||
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
|
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
|
||||||
# treated as '${Hostname}' itself on the command line.
|
# treated as '${Hostname}' itself on the command line.
|
||||||
@@ -213,7 +211,7 @@ DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
|
|||||||
set -- \
|
set -- \
|
||||||
"-Dorg.gradle.appname=$APP_BASE_NAME" \
|
"-Dorg.gradle.appname=$APP_BASE_NAME" \
|
||||||
-classpath "$CLASSPATH" \
|
-classpath "$CLASSPATH" \
|
||||||
-jar "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" \
|
org.gradle.wrapper.GradleWrapperMain \
|
||||||
"$@"
|
"$@"
|
||||||
|
|
||||||
# Stop when "xargs" is not available.
|
# Stop when "xargs" is not available.
|
||||||
1
k8s-dvfs/settings.gradle
Normal file
1
k8s-dvfs/settings.gradle
Normal file
@@ -0,0 +1 @@
|
|||||||
|
rootProject.name = 'k8s-dvfs'
|
||||||
@@ -14,10 +14,10 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import nextflow.config.scopes.Config
|
import nextflow.config.scopes.Config
|
||||||
import nextflow.k8s.client.K8sRetryConfig
|
import recreationaltech.plugin.client.K8sRetryConfig
|
||||||
|
|
||||||
import javax.annotation.Nullable
|
import javax.annotation.Nullable
|
||||||
|
|
||||||
@@ -31,13 +31,13 @@ import nextflow.config.spec.ScopeName
|
|||||||
import nextflow.container.ContainerHelper
|
import nextflow.container.ContainerHelper
|
||||||
import nextflow.script.dsl.Description
|
import nextflow.script.dsl.Description
|
||||||
import nextflow.exception.AbortOperationException
|
import nextflow.exception.AbortOperationException
|
||||||
import nextflow.k8s.client.ClientConfig
|
import recreationaltech.plugin.client.ClientConfig
|
||||||
import nextflow.k8s.client.K8sClient
|
import recreationaltech.plugin.client.K8sClient
|
||||||
import nextflow.k8s.client.K8sResponseException
|
import recreationaltech.plugin.client.K8sResponseException
|
||||||
import nextflow.k8s.model.PodOptions
|
import recreationaltech.plugin.model.PodOptions
|
||||||
import nextflow.k8s.model.PodSecurityContext
|
import recreationaltech.plugin.model.PodSecurityContext
|
||||||
import nextflow.k8s.model.PodVolumeClaim
|
import recreationaltech.plugin.model.PodVolumeClaim
|
||||||
import nextflow.k8s.model.ResourceType
|
import recreationaltech.plugin.model.ResourceType
|
||||||
import nextflow.util.Duration
|
import nextflow.util.Duration
|
||||||
|
|
||||||
import java.util.concurrent.TimeUnit
|
import java.util.concurrent.TimeUnit
|
||||||
@@ -260,6 +260,24 @@ class K8sConfig implements ConfigScope {
|
|||||||
""")
|
""")
|
||||||
final Duration noiseRuntimeEstimatorNoiseMagnitude
|
final Duration noiseRuntimeEstimatorNoiseMagnitude
|
||||||
|
|
||||||
|
@ConfigOption
|
||||||
|
@Description("""
|
||||||
|
Max. amount of time two runtime estimates can differ to be considered equal.
|
||||||
|
""")
|
||||||
|
final Duration runtimeComparisonEpsilon
|
||||||
|
|
||||||
|
@ConfigOption
|
||||||
|
@Description("""
|
||||||
|
Number of saved top runtimes used to classify a task as critical.
|
||||||
|
""")
|
||||||
|
final int dvfsSchedulingNumTopRuntimes
|
||||||
|
|
||||||
|
@ConfigOption
|
||||||
|
@Description("""
|
||||||
|
If not empty, only nodes with the given label will be used.
|
||||||
|
""")
|
||||||
|
final String nodeLabel
|
||||||
|
|
||||||
/* required by extension point -- do not remove */
|
/* required by extension point -- do not remove */
|
||||||
K8sConfig() {
|
K8sConfig() {
|
||||||
this(Collections.emptyMap())
|
this(Collections.emptyMap())
|
||||||
@@ -300,7 +318,12 @@ class K8sConfig implements ConfigScope {
|
|||||||
|
|
||||||
schedulingStrategy = opts.schedulingStrategy as String ?: "Hash"
|
schedulingStrategy = opts.schedulingStrategy as String ?: "Hash"
|
||||||
runtimeEstimator = opts.runtimeEstimator as String ?: "LinearFit"
|
runtimeEstimator = opts.runtimeEstimator as String ?: "LinearFit"
|
||||||
noiseRuntimeEstimatorNoiseMagnitude = opts.noiseRuntimeEstimatorNoiseMagnitude as Duration ?: new Duration(10, TimeUnit.SECONDS)
|
noiseRuntimeEstimatorNoiseMagnitude = opts.noiseRuntimeEstimatorNoiseMagnitude as Duration ?: new Duration(30, TimeUnit.SECONDS)
|
||||||
|
runtimeComparisonEpsilon = opts.runtimeComparisonEpsilon as Duration ?: new Duration(10, TimeUnit.SECONDS)
|
||||||
|
def numTop = opts.dvfsSchedulingNumTopRuntimes
|
||||||
|
dvfsSchedulingNumTopRuntimes = numTop != null && numTop.toString().trim() != '' ? numTop as int : 3
|
||||||
|
|
||||||
|
nodeLabel = opts.nodeLabel as String ?: ""
|
||||||
|
|
||||||
// -- shortcut to pod image pull-policy
|
// -- shortcut to pod image pull-policy
|
||||||
if( imagePullPolicy )
|
if( imagePullPolicy )
|
||||||
@@ -362,6 +385,12 @@ class K8sConfig implements ConfigScope {
|
|||||||
return DEFAULT_FUSE_PLUGIN
|
return DEFAULT_FUSE_PLUGIN
|
||||||
}
|
}
|
||||||
|
|
||||||
|
String[] getNodeLabelFilter() {
|
||||||
|
if (nodeLabel.empty)
|
||||||
|
return null
|
||||||
|
return nodeLabel.split('=', 2)
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Whenever the pod should honour the entrypoint defined by the image (default: false)
|
* Whenever the pod should honour the entrypoint defined by the image (default: false)
|
||||||
*
|
*
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import groovy.util.logging.Slf4j
|
import groovy.util.logging.Slf4j
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import groovy.transform.MapConstructor
|
import groovy.transform.MapConstructor
|
||||||
|
|
||||||
@@ -32,12 +32,12 @@ import nextflow.cli.CmdRun
|
|||||||
import nextflow.config.ConfigBuilder
|
import nextflow.config.ConfigBuilder
|
||||||
import nextflow.exception.AbortOperationException
|
import nextflow.exception.AbortOperationException
|
||||||
import nextflow.file.FileHelper
|
import nextflow.file.FileHelper
|
||||||
import nextflow.k8s.client.K8sClient
|
import recreationaltech.plugin.client.K8sClient
|
||||||
import nextflow.k8s.client.K8sResponseException
|
import recreationaltech.plugin.client.K8sResponseException
|
||||||
import nextflow.k8s.model.PodEnv
|
import recreationaltech.plugin.model.PodEnv
|
||||||
import nextflow.k8s.model.PodMountConfig
|
import recreationaltech.plugin.model.PodMountConfig
|
||||||
import nextflow.k8s.model.PodSpecBuilder
|
import recreationaltech.plugin.model.PodSpecBuilder
|
||||||
import nextflow.k8s.model.ResourceType
|
import recreationaltech.plugin.model.ResourceType
|
||||||
import nextflow.scm.AssetManager
|
import nextflow.scm.AssetManager
|
||||||
import nextflow.scm.ProviderConfig
|
import nextflow.scm.ProviderConfig
|
||||||
import nextflow.util.ConfigHelper
|
import nextflow.util.ConfigHelper
|
||||||
@@ -132,8 +132,6 @@ class K8sDriverLauncher {
|
|||||||
*/
|
*/
|
||||||
private String plugins
|
private String plugins
|
||||||
|
|
||||||
private K8sNodeInitDeployer initDeployer
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Launcher entry point. Set-up the environment and create a pod that run the Nextflow
|
* Launcher entry point. Set-up the environment and create a pod that run the Nextflow
|
||||||
* application (which in turns executed each task as a pod)
|
* application (which in turns executed each task as a pod)
|
||||||
@@ -155,7 +153,6 @@ class K8sDriverLauncher {
|
|||||||
|
|
||||||
createK8sConfigMap()
|
createK8sConfigMap()
|
||||||
|
|
||||||
initDeployer.deploy()
|
|
||||||
createK8sLauncherPod()
|
createK8sLauncherPod()
|
||||||
waitPodStart()
|
waitPodStart()
|
||||||
// login into container session
|
// login into container session
|
||||||
@@ -570,7 +567,7 @@ class K8sDriverLauncher {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* Creates and executes the nextflow driver pod
|
* Creates and executes the nextflow driver pod
|
||||||
* @return A {@link nextflow.k8s.client.K8sResponseJson} response object
|
* @return A {@link recreationaltech.plugin.client.K8sResponseJson} response object
|
||||||
*/
|
*/
|
||||||
protected createK8sLauncherPod() {
|
protected createK8sLauncherPod() {
|
||||||
final spec = makeLauncherSpec()
|
final spec = makeLauncherSpec()
|
||||||
@@ -14,11 +14,12 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import groovy.transform.CompileDynamic
|
import groovy.transform.CompileDynamic
|
||||||
import nextflow.k8s.strategies.K8sDVFSSchedulingStrategy
|
import recreationaltech.plugin.strategies.K8sDVFSFrequencyBandsSchedulingStrategy
|
||||||
import nextflow.k8s.strategies.K8sHashSchedulingStrategy
|
import recreationaltech.plugin.strategies.K8sDVFSSchedulingStrategy
|
||||||
|
import recreationaltech.plugin.strategies.K8sHashSchedulingStrategy
|
||||||
|
|
||||||
import java.util.concurrent.TimeUnit
|
import java.util.concurrent.TimeUnit
|
||||||
|
|
||||||
@@ -29,7 +30,7 @@ import groovy.transform.Memoized
|
|||||||
import groovy.util.logging.Slf4j
|
import groovy.util.logging.Slf4j
|
||||||
import nextflow.executor.Executor
|
import nextflow.executor.Executor
|
||||||
import nextflow.fusion.FusionHelper
|
import nextflow.fusion.FusionHelper
|
||||||
import nextflow.k8s.client.K8sClient
|
import recreationaltech.plugin.client.K8sClient
|
||||||
import nextflow.processor.TaskHandler
|
import nextflow.processor.TaskHandler
|
||||||
import nextflow.processor.TaskMonitor
|
import nextflow.processor.TaskMonitor
|
||||||
import nextflow.processor.TaskPollingMonitor
|
import nextflow.processor.TaskPollingMonitor
|
||||||
@@ -45,7 +46,7 @@ import org.pf4j.ExtensionPoint
|
|||||||
*/
|
*/
|
||||||
@Slf4j
|
@Slf4j
|
||||||
@CompileStatic
|
@CompileStatic
|
||||||
@ServiceName('k8s')
|
@ServiceName('k8s-dvfs')
|
||||||
class K8sExecutor extends Executor implements ExtensionPoint {
|
class K8sExecutor extends Executor implements ExtensionPoint {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -57,6 +58,8 @@ class K8sExecutor extends Executor implements ExtensionPoint {
|
|||||||
private K8sTaskScheduler taskScheduler
|
private K8sTaskScheduler taskScheduler
|
||||||
private Thread schedulerThread
|
private Thread schedulerThread
|
||||||
|
|
||||||
|
private K8sNodeInitDeployer initDeployer
|
||||||
|
|
||||||
K8sRuntimeRecorder runtimeRecorder
|
K8sRuntimeRecorder runtimeRecorder
|
||||||
K8sRuntimeEstimator runtimeEstimator
|
K8sRuntimeEstimator runtimeEstimator
|
||||||
|
|
||||||
@@ -96,6 +99,9 @@ class K8sExecutor extends Executor implements ExtensionPoint {
|
|||||||
|
|
||||||
log.debug "[K8s] config=$k8sConfig; API client config=$client.config"
|
log.debug "[K8s] config=$k8sConfig; API client config=$client.config"
|
||||||
|
|
||||||
|
this.initDeployer = new K8sNodeInitDeployer(client, k8sConfig)
|
||||||
|
initDeployer.deploy()
|
||||||
|
|
||||||
this.runtimeRecorder = new K8sRuntimeRecorder(k8sConfig.recordTaskRuntimes, k8sConfig.runtimeRecordPath)
|
this.runtimeRecorder = new K8sRuntimeRecorder(k8sConfig.recordTaskRuntimes, k8sConfig.runtimeRecordPath)
|
||||||
|
|
||||||
if (k8sConfig.runtimeEstimator == "LinearFit") {
|
if (k8sConfig.runtimeEstimator == "LinearFit") {
|
||||||
@@ -112,26 +118,31 @@ class K8sExecutor extends Executor implements ExtensionPoint {
|
|||||||
K8sSchedulingStrategy strategy = null
|
K8sSchedulingStrategy strategy = null
|
||||||
if (k8sConfig.schedulingStrategy == "Hash") {
|
if (k8sConfig.schedulingStrategy == "Hash") {
|
||||||
strategy = new K8sHashSchedulingStrategy()
|
strategy = new K8sHashSchedulingStrategy()
|
||||||
} else if (k8sConfig.schedulingStrategy == "DVFS") {
|
} else if (k8sConfig.schedulingStrategy == "DVFS" || k8sConfig.schedulingStrategy == "DVFS-SPEED") {
|
||||||
String[] ips = new String[nodes.length]
|
String[] ips = new String[nodes.length]
|
||||||
for (int i = 0; i < nodes.length; i++) {
|
for (int i = 0; i < nodes.length; i++) {
|
||||||
ips[i] = client.getPodIpAddress(K8sNodeInitDeployer.buildPodName(nodes[i]))
|
// Use node external IP since pods use hostNetwork
|
||||||
|
ips[i] = client.getNodeExternalIp(nodes[i])
|
||||||
log.info "[K8s] node ${nodes[i]} -> ${ips[i]}"
|
log.info "[K8s] node ${nodes[i]} -> ${ips[i]}"
|
||||||
}
|
}
|
||||||
strategy = new K8sDVFSSchedulingStrategy(this.runtimeEstimator,
|
strategy = new K8sDVFSSchedulingStrategy(this.runtimeEstimator,
|
||||||
new K8sDVFSClient(nodes, ips),
|
new K8sDVFSClient(nodes, ips),
|
||||||
() -> getClient())
|
() -> getClient(),
|
||||||
strategy.fullSpeedMode = false
|
k8sConfig.runtimeComparisonEpsilon,
|
||||||
} else if (k8sConfig.schedulingStrategy == "DVFS-SPEED") {
|
k8sConfig.dvfsSchedulingNumTopRuntimes)
|
||||||
|
strategy.fullSpeedMode = k8sConfig.schedulingStrategy == "DVFS-SPEED"
|
||||||
|
} else if (k8sConfig.schedulingStrategy == "DVFS-BAND") {
|
||||||
String[] ips = new String[nodes.length]
|
String[] ips = new String[nodes.length]
|
||||||
for (int i = 0; i < nodes.length; i++) {
|
for (int i = 0; i < nodes.length; i++) {
|
||||||
ips[i] = client.getPodIpAddress(K8sNodeInitDeployer.buildPodName(nodes[i]))
|
// Use node external IP since pods use hostNetwork
|
||||||
|
ips[i] = client.getNodeExternalIp(nodes[i])
|
||||||
log.info "[K8s] node ${nodes[i]} -> ${ips[i]}"
|
log.info "[K8s] node ${nodes[i]} -> ${ips[i]}"
|
||||||
}
|
}
|
||||||
strategy = new K8sDVFSSchedulingStrategy(this.runtimeEstimator,
|
strategy = new K8sDVFSFrequencyBandsSchedulingStrategy(this.runtimeEstimator,
|
||||||
new K8sDVFSClient(nodes, ips),
|
new K8sDVFSClient(nodes, ips),
|
||||||
() -> getClient())
|
() -> getClient(),
|
||||||
strategy.fullSpeedMode = true
|
k8sConfig.runtimeComparisonEpsilon,
|
||||||
|
k8sConfig.dvfsSchedulingNumTopRuntimes)
|
||||||
} else {
|
} else {
|
||||||
log.error "[K8s] invalid scheduling strategy $k8sConfig.schedulingStrategy, falling back on \"Hash\""
|
log.error "[K8s] invalid scheduling strategy $k8sConfig.schedulingStrategy, falling back on \"Hash\""
|
||||||
strategy = new K8sHashSchedulingStrategy()
|
strategy = new K8sHashSchedulingStrategy()
|
||||||
@@ -144,10 +155,19 @@ class K8sExecutor extends Executor implements ExtensionPoint {
|
|||||||
|
|
||||||
@CompileDynamic
|
@CompileDynamic
|
||||||
private String[] getNodeList() {
|
private String[] getNodeList() {
|
||||||
final resp = getClient().nodeList()
|
final resp = client.nodeList()
|
||||||
ArrayList<String> nodes = new ArrayList<String>()
|
ArrayList<String> nodes = new ArrayList<String>()
|
||||||
for ( Map item : resp.items ) {
|
String[] filter = getK8sConfig().getNodeLabelFilter()
|
||||||
nodes.add(item.metadata.name as String)
|
if (filter == null) {
|
||||||
|
for (Map item : resp.items) {
|
||||||
|
nodes.add(item.metadata.name as String)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
for (Map item : resp.items) {
|
||||||
|
Map<String, String> labels = item.metadata.labels as Map<String, String>
|
||||||
|
if (labels.get(filter[0]) == filter[1])
|
||||||
|
nodes.add(item.metadata.name as String)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return nodes.toArray()
|
return nodes.toArray()
|
||||||
}
|
}
|
||||||
@@ -157,6 +177,7 @@ class K8sExecutor extends Executor implements ExtensionPoint {
|
|||||||
this.runtimeRecorder.write()
|
this.runtimeRecorder.write()
|
||||||
this.taskScheduler.stop()
|
this.taskScheduler.stop()
|
||||||
this.schedulerThread.join()
|
this.schedulerThread.join()
|
||||||
|
this.initDeployer.cleanup()
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import groovy.util.logging.Slf4j
|
import groovy.util.logging.Slf4j
|
||||||
|
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import groovy.util.logging.Slf4j
|
import groovy.util.logging.Slf4j
|
||||||
import nextflow.config.scopes.Config
|
import nextflow.config.scopes.Config
|
||||||
@@ -1,9 +1,9 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import groovy.util.logging.Slf4j
|
import groovy.util.logging.Slf4j
|
||||||
import nextflow.k8s.client.K8sClient
|
import recreationaltech.plugin.client.K8sClient
|
||||||
import nextflow.k8s.model.PodHostMount
|
import recreationaltech.plugin.model.PodHostMount
|
||||||
import nextflow.k8s.model.PodSpecBuilder
|
import recreationaltech.plugin.model.PodSpecBuilder
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
class K8sNodeInitDeployer {
|
class K8sNodeInitDeployer {
|
||||||
@@ -19,13 +19,14 @@ class K8sNodeInitDeployer {
|
|||||||
final init = config.nodeInit
|
final init = config.nodeInit
|
||||||
if ( !init?.enabled )
|
if ( !init?.enabled )
|
||||||
return
|
return
|
||||||
|
final namespace = config.namespace
|
||||||
|
|
||||||
log.info("deploying init pods")
|
log.info("deploying init pods")
|
||||||
|
|
||||||
final nodes = getNodes()
|
final nodes = getNodes()
|
||||||
for ( String nodeName : nodes ) {
|
for ( String nodeName : nodes ) {
|
||||||
log.info(" ... deploying to " + nodeName)
|
log.info(" ... deploying to " + nodeName)
|
||||||
final spec = makePodSpec(init, nodeName)
|
final spec = makePodSpec(init, nodeName, namespace)
|
||||||
client.podCreate(spec)
|
client.podCreate(spec)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -108,8 +109,17 @@ exit 0
|
|||||||
private List<String> getNodes() {
|
private List<String> getNodes() {
|
||||||
final resp = client.nodeList()
|
final resp = client.nodeList()
|
||||||
ArrayList<String> nodes = new ArrayList<String>()
|
ArrayList<String> nodes = new ArrayList<String>()
|
||||||
for ( Map item: resp.items ) {
|
String[] filter = config.getNodeLabelFilter()
|
||||||
nodes.add(item.metadata.name as String)
|
if (filter == null) {
|
||||||
|
for (Map item : resp.items) {
|
||||||
|
nodes.add(item.metadata.name as String)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
for (Map item : resp.items) {
|
||||||
|
Map<String, String> labels = item.metadata.labels as Map<String, String>
|
||||||
|
if (labels.get(filter[0]) == filter[1])
|
||||||
|
nodes.add(item.metadata.name as String)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return nodes
|
return nodes
|
||||||
}
|
}
|
||||||
@@ -123,7 +133,7 @@ exit 0
|
|||||||
return name
|
return name
|
||||||
}
|
}
|
||||||
|
|
||||||
private Map makePodSpec(K8sNodeInitConfig config, String nodeName) {
|
private Map makePodSpec(K8sNodeInitConfig config, String nodeName, String namespace) {
|
||||||
ArrayList<PodHostMount> mounts = new ArrayList<PodHostMount>()
|
ArrayList<PodHostMount> mounts = new ArrayList<PodHostMount>()
|
||||||
mounts.add(new PodHostMount("/sys", "/sys"))
|
mounts.add(new PodHostMount("/sys", "/sys"))
|
||||||
mounts.add(new PodHostMount("/dev", "/dev"))
|
mounts.add(new PodHostMount("/dev", "/dev"))
|
||||||
@@ -137,6 +147,8 @@ exit 0
|
|||||||
.withHostMounts(mounts)
|
.withHostMounts(mounts)
|
||||||
.withPodName(buildPodName(nodeName))
|
.withPodName(buildPodName(nodeName))
|
||||||
.withPort(8080)
|
.withPort(8080)
|
||||||
|
.withHostNetwork(true)
|
||||||
|
.withNamespace(namespace)
|
||||||
.build()
|
.build()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import nextflow.util.Duration
|
import nextflow.util.Duration
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import nextflow.plugin.BasePlugin
|
import nextflow.plugin.BasePlugin
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import groovy.util.logging.Slf4j
|
import groovy.util.logging.Slf4j
|
||||||
|
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
record K8sRuntimeRecord(
|
record K8sRuntimeRecord(
|
||||||
String taskName,
|
String taskName,
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import groovy.util.logging.Slf4j
|
import groovy.util.logging.Slf4j
|
||||||
|
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
|
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import nextflow.processor.TaskRun
|
import nextflow.processor.TaskRun
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
interface K8sSchedulingStrategy {
|
interface K8sSchedulingStrategy {
|
||||||
/**
|
/**
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import java.nio.file.FileAlreadyExistsException
|
import java.nio.file.FileAlreadyExistsException
|
||||||
import java.nio.file.Files
|
import java.nio.file.Files
|
||||||
@@ -29,16 +29,16 @@ import nextflow.SysEnv
|
|||||||
import nextflow.container.ContainerHelper
|
import nextflow.container.ContainerHelper
|
||||||
import nextflow.container.DockerBuilder
|
import nextflow.container.DockerBuilder
|
||||||
import nextflow.exception.NodeTerminationException
|
import nextflow.exception.NodeTerminationException
|
||||||
import nextflow.k8s.client.PodUnschedulableException
|
import recreationaltech.plugin.client.PodUnschedulableException
|
||||||
import nextflow.exception.ProcessSubmitException
|
import nextflow.exception.ProcessSubmitException
|
||||||
import nextflow.executor.BashWrapperBuilder
|
import nextflow.executor.BashWrapperBuilder
|
||||||
import nextflow.fusion.FusionAwareTask
|
import nextflow.fusion.FusionAwareTask
|
||||||
import nextflow.k8s.client.K8sClient
|
import recreationaltech.plugin.client.K8sClient
|
||||||
import nextflow.k8s.client.K8sResponseException
|
import recreationaltech.plugin.client.K8sResponseException
|
||||||
import nextflow.k8s.model.PodEnv
|
import recreationaltech.plugin.model.PodEnv
|
||||||
import nextflow.k8s.model.PodOptions
|
import recreationaltech.plugin.model.PodOptions
|
||||||
import nextflow.k8s.model.PodSpecBuilder
|
import recreationaltech.plugin.model.PodSpecBuilder
|
||||||
import nextflow.k8s.model.ResourceType
|
import recreationaltech.plugin.model.ResourceType
|
||||||
import nextflow.processor.TaskHandler
|
import nextflow.processor.TaskHandler
|
||||||
import nextflow.processor.TaskRun
|
import nextflow.processor.TaskRun
|
||||||
import nextflow.processor.TaskStatus
|
import nextflow.processor.TaskStatus
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
@@ -14,11 +14,11 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.cli
|
package recreationaltech.plugin.cli
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import nextflow.cli.CmdKubeRun
|
import nextflow.cli.CmdKubeRun
|
||||||
import nextflow.k8s.K8sDriverLauncher
|
import recreationaltech.plugin.K8sDriverLauncher
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Kuberun command implementation logic
|
* Kuberun command implementation logic
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import groovy.util.logging.Slf4j
|
import groovy.util.logging.Slf4j
|
||||||
import nextflow.util.Duration
|
import nextflow.util.Duration
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import javax.net.ssl.KeyManager
|
import javax.net.ssl.KeyManager
|
||||||
import javax.net.ssl.KeyManagerFactory
|
import javax.net.ssl.KeyManagerFactory
|
||||||
@@ -14,13 +14,14 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import dev.failsafe.Failsafe
|
import dev.failsafe.Failsafe
|
||||||
import dev.failsafe.FailsafeException
|
import dev.failsafe.FailsafeException
|
||||||
import dev.failsafe.RetryPolicy
|
import dev.failsafe.RetryPolicy
|
||||||
import dev.failsafe.event.EventListener
|
import dev.failsafe.event.EventListener
|
||||||
import dev.failsafe.event.ExecutionAttemptedEvent
|
import dev.failsafe.event.ExecutionAttemptedEvent
|
||||||
|
import dev.failsafe.function.CheckedPredicate
|
||||||
import dev.failsafe.function.CheckedSupplier
|
import dev.failsafe.function.CheckedSupplier
|
||||||
import nextflow.exception.K8sOutOfCpuException
|
import nextflow.exception.K8sOutOfCpuException
|
||||||
import nextflow.exception.K8sOutOfMemoryException
|
import nextflow.exception.K8sOutOfMemoryException
|
||||||
@@ -371,6 +372,32 @@ class K8sClient {
|
|||||||
(resp?.spec as Map)?.nodeName as String
|
(resp?.spec as Map)?.nodeName as String
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get the external IP address of a node
|
||||||
|
* @param nodeName The node name
|
||||||
|
* @return The external IP address of the node, or internal IP if external is not available
|
||||||
|
*/
|
||||||
|
String getNodeExternalIp(String nodeName) {
|
||||||
|
assert nodeName
|
||||||
|
final K8sResponseJson resp = nodeDescribe(nodeName)
|
||||||
|
final status = resp.status as Map
|
||||||
|
final addresses = status?.addresses as List<Map>
|
||||||
|
|
||||||
|
// Try to find ExternalIP first
|
||||||
|
for (Map address : addresses) {
|
||||||
|
if (address.type == 'ExternalIP') {
|
||||||
|
return address.address as String
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Fall back to InternalIP
|
||||||
|
for (Map address : addresses) {
|
||||||
|
if (address.type == 'InternalIP') {
|
||||||
|
return address.address as String
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get pod current state object
|
* Get pod current state object
|
||||||
*
|
*
|
||||||
@@ -870,10 +897,17 @@ class K8sClient {
|
|||||||
trace('GET', action, resp.text)
|
trace('GET', action, resp.text)
|
||||||
final podList = new K8sResponseJson(resp.text)
|
final podList = new K8sResponseJson(resp.text)
|
||||||
final items = podList.items as List<Map>
|
final items = podList.items as List<Map>
|
||||||
|
|
||||||
long totalAllocated = 0L
|
long totalAllocated = 0L
|
||||||
if (items) {
|
if (items) {
|
||||||
for (Map pod : items) {
|
for (Map pod : items) {
|
||||||
|
// Skip terminated/succeeded pods
|
||||||
|
final status = pod.status as Map
|
||||||
|
final phase = status?.phase as String
|
||||||
|
if (phase in ['Succeeded', 'Failed', 'Unknown']) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
final spec = pod.spec as Map
|
final spec = pod.spec as Map
|
||||||
if (spec) {
|
if (spec) {
|
||||||
final containers = spec.containers as List<Map>
|
final containers = spec.containers as List<Map>
|
||||||
@@ -936,6 +970,13 @@ class K8sClient {
|
|||||||
long totalAllocated = 0L
|
long totalAllocated = 0L
|
||||||
if (items) {
|
if (items) {
|
||||||
for (Map pod : items) {
|
for (Map pod : items) {
|
||||||
|
// Skip terminated/succeeded pods
|
||||||
|
final status = pod.status as Map
|
||||||
|
final phase = status?.phase as String
|
||||||
|
if (phase in ['Succeeded', 'Failed', 'Unknown']) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
final spec = pod.spec as Map
|
final spec = pod.spec as Map
|
||||||
if (spec) {
|
if (spec) {
|
||||||
final containers = spec.containers as List<Map>
|
final containers = spec.containers as List<Map>
|
||||||
@@ -1187,17 +1228,17 @@ class K8sClient {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Creates a retry policy using the configuration specified by {@link nextflow.k8s.client.K8sRetryConfig}
|
* Creates a retry policy using the configuration specified by {@link recreationaltech.plugin.client.K8sRetryConfig}
|
||||||
*
|
*
|
||||||
* @param cond A predicate that determines when a retry should be triggered
|
* @param cond A predicate that determines when a retry should be triggered
|
||||||
* @return The {@link dev.failsafe.RetryPolicy} instance
|
* @return The {@link dev.failsafe.RetryPolicy} instance
|
||||||
*/
|
*/
|
||||||
protected <T> RetryPolicy<T> retryPolicy(Predicate<? extends Throwable> cond) {
|
protected <T> RetryPolicy<T> retryPolicy(CheckedPredicate<? extends Throwable> cond) {
|
||||||
final cfg = config.retryConfig
|
final cfg = config.retryConfig
|
||||||
final listener = new EventListener<ExecutionAttemptedEvent<T>>() {
|
final listener = new EventListener<ExecutionAttemptedEvent<T>>() {
|
||||||
@Override
|
@Override
|
||||||
void accept(ExecutionAttemptedEvent<T> event) throws Throwable {
|
void accept(ExecutionAttemptedEvent<T> event) throws Throwable {
|
||||||
log.debug("K8s response error - attempt: ${event.attemptCount}; reason: ${event.lastFailure.message}")
|
log.debug("K8s response error - attempt: ${event.attemptCount}; reason: ${event.lastException?.message ?: 'unknown'}")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return RetryPolicy.<T>builder()
|
return RetryPolicy.<T>builder()
|
||||||
@@ -1219,9 +1260,8 @@ class K8sClient {
|
|||||||
*/
|
*/
|
||||||
protected <T> T apply(CheckedSupplier<T> action) {
|
protected <T> T apply(CheckedSupplier<T> action) {
|
||||||
// define the retry condition
|
// define the retry condition
|
||||||
final cond = new Predicate<? extends Throwable>() {
|
final cond = [
|
||||||
@Override
|
test: { Throwable t ->
|
||||||
boolean test(Throwable t) {
|
|
||||||
if ( t instanceof K8sResponseException && t.response.code in RETRY_CODES )
|
if ( t instanceof K8sResponseException && t.response.code in RETRY_CODES )
|
||||||
return true
|
return true
|
||||||
if( t instanceof SocketException || t.cause instanceof SocketException )
|
if( t instanceof SocketException || t.cause instanceof SocketException )
|
||||||
@@ -1230,7 +1270,7 @@ class K8sClient {
|
|||||||
return true
|
return true
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
}
|
] as CheckedPredicate<Throwable>
|
||||||
// create the retry policy object
|
// create the retry policy object
|
||||||
final policy = retryPolicy(cond)
|
final policy = retryPolicy(cond)
|
||||||
// apply the action with and throw the original cause
|
// apply the action with and throw the original cause
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import groovy.util.logging.Slf4j
|
import groovy.util.logging.Slf4j
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import groovy.json.JsonOutput
|
import groovy.json.JsonOutput
|
||||||
import groovy.json.JsonSlurper
|
import groovy.json.JsonSlurper
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import groovy.transform.EqualsAndHashCode
|
import groovy.transform.EqualsAndHashCode
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import nextflow.exception.ProcessException
|
import nextflow.exception.ProcessException
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client;
|
package recreationaltech.plugin.client;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* This file is derived from
|
* This file is derived from
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import groovy.transform.EqualsAndHashCode
|
import groovy.transform.EqualsAndHashCode
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import groovy.transform.EqualsAndHashCode
|
import groovy.transform.EqualsAndHashCode
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import java.nio.file.Paths
|
import java.nio.file.Paths
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import java.nio.file.Paths
|
import java.nio.file.Paths
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import groovy.transform.EqualsAndHashCode
|
import groovy.transform.EqualsAndHashCode
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import java.nio.file.Paths
|
import java.nio.file.Paths
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import groovy.transform.EqualsAndHashCode
|
import groovy.transform.EqualsAndHashCode
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import groovy.transform.EqualsAndHashCode
|
import groovy.transform.EqualsAndHashCode
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import groovy.transform.EqualsAndHashCode
|
import groovy.transform.EqualsAndHashCode
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import java.nio.file.Path
|
import java.nio.file.Path
|
||||||
import java.util.concurrent.atomic.AtomicInteger
|
import java.util.concurrent.atomic.AtomicInteger
|
||||||
@@ -130,6 +130,8 @@ class PodSpecBuilder {
|
|||||||
|
|
||||||
Integer port = null
|
Integer port = null
|
||||||
|
|
||||||
|
boolean hostNetwork = false
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @return A sequential volume unique identifier
|
* @return A sequential volume unique identifier
|
||||||
*/
|
*/
|
||||||
@@ -413,6 +415,11 @@ class PodSpecBuilder {
|
|||||||
return this
|
return this
|
||||||
}
|
}
|
||||||
|
|
||||||
|
PodSpecBuilder withHostNetwork(boolean value) {
|
||||||
|
this.hostNetwork = value
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
@PackageScope List<Map> createPullSecret() {
|
@PackageScope List<Map> createPullSecret() {
|
||||||
def result = new ArrayList(1)
|
def result = new ArrayList(1)
|
||||||
def entry = new LinkedHashMap(1)
|
def entry = new LinkedHashMap(1)
|
||||||
@@ -519,6 +526,9 @@ class PodSpecBuilder {
|
|||||||
if ( nodeName )
|
if ( nodeName )
|
||||||
spec.nodeName = nodeName
|
spec.nodeName = nodeName
|
||||||
|
|
||||||
|
if ( hostNetwork )
|
||||||
|
spec.hostNetwork = true
|
||||||
|
|
||||||
final pod = [
|
final pod = [
|
||||||
apiVersion: 'v1',
|
apiVersion: 'v1',
|
||||||
kind: 'Pod',
|
kind: 'Pod',
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import groovy.transform.EqualsAndHashCode
|
import groovy.transform.EqualsAndHashCode
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Model the resource type to be used to run nextflow tasks
|
* Model the resource type to be used to run nextflow tasks
|
||||||
@@ -0,0 +1,395 @@
|
|||||||
|
package recreationaltech.plugin.strategies
|
||||||
|
|
||||||
|
import groovy.transform.CompileStatic
|
||||||
|
import groovy.util.logging.Slf4j
|
||||||
|
import nextflow.processor.TaskRun
|
||||||
|
import nextflow.util.Duration
|
||||||
|
import recreationaltech.plugin.*
|
||||||
|
import recreationaltech.plugin.client.K8sClient
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Implements a scheduling strategy utilizing dvfs to reduce the energy consumption
|
||||||
|
* of workflow execution, while attempting to maintain the same makespan.
|
||||||
|
*
|
||||||
|
* This variant divides available nodes into three classes:
|
||||||
|
* - High frequency (100%)
|
||||||
|
* - Middle frequency (66%)
|
||||||
|
* - Low frequency (33%)
|
||||||
|
*
|
||||||
|
* Tasks are assigned to frequency classes according to their
|
||||||
|
* estimated runtime t:
|
||||||
|
* - t is in top-k runtimes => High
|
||||||
|
* - t is at most 1/2 top runtime => Middle
|
||||||
|
* - t is below 1/2 top runtime => Low
|
||||||
|
*
|
||||||
|
* Nodes are assigned to the classes via round robin,
|
||||||
|
* starting with high.
|
||||||
|
*/
|
||||||
|
@Slf4j
|
||||||
|
@CompileStatic
|
||||||
|
class K8sDVFSFrequencyBandsSchedulingStrategy implements K8sSchedulingStrategy {
|
||||||
|
|
||||||
|
/** Used for passing K8sExecutor.getClient.
|
||||||
|
* We cannot pass the client directly, becaue it can be refreshed
|
||||||
|
* during workflow execution.
|
||||||
|
*/
|
||||||
|
public interface K8sClientGetter {
|
||||||
|
K8sClient getClient()
|
||||||
|
}
|
||||||
|
|
||||||
|
private static long getTaskMemoryRequirment(TaskRun task) {
|
||||||
|
return task.config.getMemory() ? task.config.getMemory().bytes : 64 * 1024 * 1024
|
||||||
|
}
|
||||||
|
|
||||||
|
private static int getTaskCPURequirement(TaskRun task) {
|
||||||
|
return task.config.hasCpus() ? task.config.getCpus() * 1000 : 1000
|
||||||
|
}
|
||||||
|
|
||||||
|
@Slf4j
|
||||||
|
private class WorkerNode {
|
||||||
|
private class AssignedTask {
|
||||||
|
TaskRun task
|
||||||
|
|
||||||
|
AssignedTask(TaskRun t) {
|
||||||
|
this.task = t
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
String name
|
||||||
|
|
||||||
|
long frequency
|
||||||
|
|
||||||
|
ArrayList<AssignedTask> tasks
|
||||||
|
|
||||||
|
WorkerNode(String name, long frequency, K8sClientGetter clientGetter) {
|
||||||
|
this.name = name
|
||||||
|
this.frequency = frequency
|
||||||
|
this.tasks = new ArrayList<>()
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Return the number of available (unoccupied) bytes */
|
||||||
|
long getAvailableMemory() {
|
||||||
|
def k8sClient = clientGetter.getClient()
|
||||||
|
Long available = k8sClient.getNodeMemoryAvailableBytes(this.name)
|
||||||
|
if (available == null) {
|
||||||
|
log.warn "[K8s] failed to retrieve available memory for node ${name}"
|
||||||
|
// Fallback: try capacity - allocated
|
||||||
|
Long capacity = k8sClient.getNodeMemoryCapacityBytes(this.name)
|
||||||
|
Long allocated = k8sClient.getNodeMemoryAllocatedBytes(this.name)
|
||||||
|
if (capacity != null && allocated != null) {
|
||||||
|
return capacity - allocated
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return available
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Return the number of available (unoccupied) cpu cores */
|
||||||
|
long getAvailableCPUs() {
|
||||||
|
def k8sClient = clientGetter.getClient()
|
||||||
|
Long available = k8sClient.getNodeCpuAvailableMillis(this.name)
|
||||||
|
if (available == null) {
|
||||||
|
log.warn "[K8s] failed to retrieve available CPU for node ${name}"
|
||||||
|
// Fallback: try capacity - allocated
|
||||||
|
Long capacity = k8sClient.getNodeCpuCapacityMillis(this.name)
|
||||||
|
Long allocated = k8sClient.getNodeCpuAllocatedMillis(this.name)
|
||||||
|
if (capacity != null && allocated != null) {
|
||||||
|
return capacity - allocated
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return available
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Return the total amount of installed memory */
|
||||||
|
long getMemoryAmount() {
|
||||||
|
def k8sClient = clientGetter.getClient()
|
||||||
|
Long capacity = k8sClient.getNodeMemoryCapacityBytes(this.name)
|
||||||
|
return capacity != null ? capacity.longValue() : 0
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Return the total number of installed cpu cores */
|
||||||
|
long getCPUCount() {
|
||||||
|
def k8sClient = clientGetter.getClient()
|
||||||
|
Long capacity = k8sClient.getNodeCpuCapacityMillis(this.name)
|
||||||
|
return capacity != null ? capacity.longValue(): 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// Resets the frequency
|
||||||
|
private void updateFrequency(K8sDVFSClient dvfsClient) {
|
||||||
|
if (tasks.size() == 0)
|
||||||
|
return
|
||||||
|
long maxFrequency = dvfsClient.getNodeMaxFrequency(this.name).orElse(this.frequency)
|
||||||
|
log.info "[K8s] node ${name} running at ${frequency} Hz / ${maxFrequency} Hz ${((double)frequency/(double)maxFrequency) * 100.0}%"
|
||||||
|
dvfsClient.setNodeFrequency(name, (int)frequency)
|
||||||
|
}
|
||||||
|
|
||||||
|
void assignTask(TaskRun task, K8sDVFSClient dvfsClient) {
|
||||||
|
log.info "[K8s] node ${name}: task ${task.name} assigned with ${this.frequency}"
|
||||||
|
this.tasks.add(new AssignedTask(task))
|
||||||
|
updateFrequency(dvfsClient)
|
||||||
|
}
|
||||||
|
|
||||||
|
void taskFinished(TaskRun task, K8sDVFSClient dvfsClient) {
|
||||||
|
log.info "[K8s] node ${name}: task ${task.name} finished"
|
||||||
|
this.tasks.removeIf {it.task == task}
|
||||||
|
updateFrequency(dvfsClient)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
class SchedulingRequestComparator implements Comparator<K8sSchedulingRequest> {
|
||||||
|
K8sRuntimeEstimator runtimeEstimator
|
||||||
|
long currentTime
|
||||||
|
double epsilon
|
||||||
|
|
||||||
|
@Override
|
||||||
|
int compare(K8sSchedulingRequest o1, K8sSchedulingRequest o2) {
|
||||||
|
// First, check if one of the tasks is (estimated to be) on the critical path
|
||||||
|
double t1 = runtimeEstimator.estimate(o1.handler)
|
||||||
|
double t2 = runtimeEstimator.estimate(o2.handler)
|
||||||
|
|
||||||
|
if (t1 > t2 + epsilon)
|
||||||
|
return -1
|
||||||
|
else if (t2 > t1 + epsilon)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
// Both are not on the critical path. Sort based on the time they spent in the queue
|
||||||
|
long w1 = currentTime - o1.submitTimeMillis
|
||||||
|
long w2 = currentTime - o2.submitTimeMillis
|
||||||
|
if (w1 > w2)
|
||||||
|
return -1
|
||||||
|
else if (w2 > w1)
|
||||||
|
return 1
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private K8sRuntimeEstimator runtimeEstimator
|
||||||
|
private K8sDVFSClient dvfsClient
|
||||||
|
|
||||||
|
private ArrayList<WorkerNode> nodes
|
||||||
|
private HashMap<String, WorkerNode> taskToNode
|
||||||
|
private long globalMaxFrequency
|
||||||
|
private long globalMinFrequency
|
||||||
|
private long[] frequencies
|
||||||
|
|
||||||
|
private K8sClientGetter clientGetter
|
||||||
|
|
||||||
|
private double comparisonEpsilonMillis
|
||||||
|
|
||||||
|
private double[] topRuntimes
|
||||||
|
private double averageRuntime
|
||||||
|
private double finishedTaskCount
|
||||||
|
|
||||||
|
K8sDVFSFrequencyBandsSchedulingStrategy(K8sRuntimeEstimator runtimeEstimator,
|
||||||
|
K8sDVFSClient dvfsClient,
|
||||||
|
K8sClientGetter clientGetter,
|
||||||
|
Duration runtimeComparisonEpsilon,
|
||||||
|
int topRuntimeCount) {
|
||||||
|
this.runtimeEstimator = runtimeEstimator
|
||||||
|
this.dvfsClient = dvfsClient
|
||||||
|
this.nodes = new ArrayList<>()
|
||||||
|
this.taskToNode = new HashMap<>();
|
||||||
|
this.clientGetter = clientGetter
|
||||||
|
this.comparisonEpsilonMillis = (double)runtimeComparisonEpsilon.toMillis()
|
||||||
|
this.topRuntimes = new double[topRuntimeCount]
|
||||||
|
for (int i = 0; i < topRuntimeCount; i++) {
|
||||||
|
this.topRuntimes[i] = 0.0
|
||||||
|
}
|
||||||
|
this.averageRuntime = 0.0
|
||||||
|
this.finishedTaskCount = 0.0
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean isInTopRuntimes(double rt) {
|
||||||
|
for (int i = 0; i < topRuntimes.size(); i++) {
|
||||||
|
if (rt >= topRuntimes[i])
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
private void updateTopRuntimes(double rt) {
|
||||||
|
for (int i = 0; i < topRuntimes.size(); i++) {
|
||||||
|
if (rt > topRuntimes[i]) {
|
||||||
|
/* Move all one down */
|
||||||
|
for (int j = topRuntimes.size() - 1; j > i; j--) {
|
||||||
|
topRuntimes[j] = topRuntimes[j - 1];
|
||||||
|
}
|
||||||
|
topRuntimes[i] = rt
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
K8sSchedulingDecision schedule(K8sTaskScheduler scheduler, List<K8sSchedulingRequest> queue) {
|
||||||
|
if (nodes.isEmpty()) {
|
||||||
|
if (!initNodes(scheduler))
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Step 1: Sort by task priority. We will attempt to schedule tasks "in order", so that the
|
||||||
|
* highest priority tasks are assigned to nodes as soon as possible.
|
||||||
|
*
|
||||||
|
* Priority is based on a) the tasks estimated runtime and b) the wait time of the task.
|
||||||
|
*/
|
||||||
|
SchedulingRequestComparator comparator = new SchedulingRequestComparator()
|
||||||
|
comparator.runtimeEstimator = runtimeEstimator
|
||||||
|
comparator.epsilon = comparisonEpsilonMillis
|
||||||
|
comparator.currentTime = System.currentTimeMillis()
|
||||||
|
queue.sort(comparator)
|
||||||
|
|
||||||
|
/* Step 2: For each task attempt to schedule it onto a node */
|
||||||
|
for (K8sSchedulingRequest req : queue) {
|
||||||
|
/* Step 2.1: Determine if the task is on the critical path.
|
||||||
|
* If yes, we just schedule it at max frequency on the node with the highest available
|
||||||
|
* frequency. If not, we determine a frequency (see below).
|
||||||
|
*/
|
||||||
|
final double taskEstimation = runtimeEstimator.estimate(req.handler)
|
||||||
|
final boolean isCriticalPath = isInTopRuntimes(taskEstimation)
|
||||||
|
long frequency = this.frequencies[0]
|
||||||
|
if (!isCriticalPath) {
|
||||||
|
if (taskEstimation >= topRuntimes[0] * .5)
|
||||||
|
frequency = this.frequencies[1]
|
||||||
|
else
|
||||||
|
frequency = this.frequencies[2]
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Step 2.2: Filter nodes based on task requirements */
|
||||||
|
ArrayList<WorkerNode> suitableNodes = filterNodes(req.task)
|
||||||
|
if (suitableNodes.size() == 0) {
|
||||||
|
if (!anyNode(req.task)) {
|
||||||
|
log.error "[K8s] unable to schedule task ${req.task} - no node satisfies resource requirements ${getTaskMemoryRequirment(req.task)} bytes ${getTaskCPURequirement(req.task)} cpus"
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
/* No node can currently execute this task, but it should be possible in the future */
|
||||||
|
/* log.info "[K8s] ${req.task} can not be scheduled: ${getTaskMemoryRequirment(req.task)} bytes ${getTaskCPURequirement(req.task)} CPUs"
|
||||||
|
for (WorkerNode n : this.nodes) {
|
||||||
|
log.info "[K8s] node ${n.name} - ${n.availableMemory}, ${n.availableCPUs}"
|
||||||
|
} */
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Step 2.3: Assign to node based on "best fit" - current node frequency is closest to determined frequency */
|
||||||
|
long minDist = Math.abs(suitableNodes[0].frequency - frequency)
|
||||||
|
WorkerNode closest = suitableNodes[0]
|
||||||
|
for (WorkerNode node : suitableNodes) {
|
||||||
|
long dist = Math.abs(node.frequency - frequency)
|
||||||
|
if (dist < minDist) {
|
||||||
|
closest = node
|
||||||
|
minDist = dist
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
closest.assignTask(req.task, dvfsClient)
|
||||||
|
taskToNode.put(req.task.hash.toString(), closest)
|
||||||
|
log.info "[K8s] DVFS: Assigned task ${req.task} to node ${closest.name} - ${taskToNode.size()} assigned tasks"
|
||||||
|
return new K8sSchedulingDecision(req, closest.name)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.info "[K8s] unable to schedule any task. The queue contains ${queue.size()} tasks."
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
boolean scheduleImmediately(K8sTaskScheduler scheduler, List<K8sSchedulingRequest> queue) {
|
||||||
|
if (nodes.isEmpty()) {
|
||||||
|
if (!initNodes(scheduler))
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
/* We want to schedule immediately if there are unoccupied nodes */
|
||||||
|
boolean doIt = queue != null && queue.size() > 0 && taskToNode.size() < nodes.size()
|
||||||
|
log.info "[K8s] scheduleImmediately: ${queue.size()} tasks in queue, ${taskToNode.size()} tasks running on ${nodes.size()} nodes: ${doIt}"
|
||||||
|
return doIt
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
synchronized void taskFinished(K8sTaskHandler task) {
|
||||||
|
/* TODO: This just uses elapsed wall-clock time, regardless of the frequency used to execute the task.
|
||||||
|
* This will skew the average towards longer runtimes, which is undesirable, because it will lead to more
|
||||||
|
* tasks classified as "critical path".
|
||||||
|
* A simple (rough) solution could be to keep track of the tasks "relative" frequency and just scale the
|
||||||
|
* elapsed time based on that.
|
||||||
|
*/
|
||||||
|
double runtime = (double)(task.getCompleteTimeMillis() - task.getStartTimeMillis())
|
||||||
|
averageRuntime = (runtime + finishedTaskCount * averageRuntime) / (finishedTaskCount + 1.0)
|
||||||
|
finishedTaskCount += 1.0
|
||||||
|
updateTopRuntimes(runtime)
|
||||||
|
|
||||||
|
/* Free resources allocated by this task */
|
||||||
|
WorkerNode node = taskToNode.get(task.task.hash.toString())
|
||||||
|
if (node != null) {
|
||||||
|
node.taskFinished(task.task, dvfsClient)
|
||||||
|
taskToNode.remove(task.task.hash.toString())
|
||||||
|
} else {
|
||||||
|
log.warn "[K8s] no node recorded for task ${task.toString()}"
|
||||||
|
}
|
||||||
|
|
||||||
|
log.info "[K8s] task ${task.toString()} finished - ${taskToNode.size()} tasks running"
|
||||||
|
if (node != null) {
|
||||||
|
log.info "[K8s] task ran on node ${node.name} - ${node.availableMemory} bytes ${node.availableCPUs}"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private synchronized boolean initNodes(K8sTaskScheduler scheduler) {
|
||||||
|
this.globalMaxFrequency = Long.MAX_VALUE
|
||||||
|
this.globalMinFrequency = Long.MIN_VALUE
|
||||||
|
final nodes = scheduler.getNodes()
|
||||||
|
|
||||||
|
for (String node : nodes) {
|
||||||
|
final min = dvfsClient.getNodeMinFrequency(node)
|
||||||
|
final max = dvfsClient.getNodeMaxFrequency(node)
|
||||||
|
|
||||||
|
if (min.empty || max.empty) {
|
||||||
|
log.error "[K8s] failed to query node $node information"
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
globalMaxFrequency = Long.min(globalMaxFrequency, max.asLong)
|
||||||
|
globalMinFrequency = Long.max(globalMinFrequency, min.asLong)
|
||||||
|
|
||||||
|
log.info "[K8s] node ${node}: ${min.asLong} Hz - ${max.asLong} Hz"
|
||||||
|
}
|
||||||
|
|
||||||
|
long diff = globalMaxFrequency - globalMinFrequency
|
||||||
|
long diff3 = diff.intdiv(3)
|
||||||
|
this.frequencies = [
|
||||||
|
globalMaxFrequency,
|
||||||
|
globalMinFrequency + diff3 * 2,
|
||||||
|
globalMinFrequency + diff3,
|
||||||
|
]
|
||||||
|
|
||||||
|
int next = 0
|
||||||
|
for (String node : nodes) {
|
||||||
|
this.nodes.add(new WorkerNode(node, this.frequencies[next], this.clientGetter))
|
||||||
|
log.info "[K8s] node ${node}: set to ${this.frequencies[next]} Hz"
|
||||||
|
next = (next + 1) % this.frequencies.length
|
||||||
|
}
|
||||||
|
|
||||||
|
return !this.nodes.isEmpty()
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Returns a list of nodes that fulfill the tasks resource requirements
|
||||||
|
*/
|
||||||
|
private ArrayList<WorkerNode> filterNodes(TaskRun task) {
|
||||||
|
final long reqBytes = getTaskMemoryRequirment(task)
|
||||||
|
final int reqCPUs = getTaskCPURequirement(task)
|
||||||
|
ArrayList<WorkerNode> suitableNodes = new ArrayList<>()
|
||||||
|
for (WorkerNode n : nodes) {
|
||||||
|
if (n.availableMemory >= reqBytes && n.availableCPUs >= reqCPUs) {
|
||||||
|
log.info "[K8s] task ${task.name}: ${reqBytes} bytes ${reqCPUs} cpus: node ${n.name} has ${n.availableMemory} bytes, ${n.availableCPUs} cpus"
|
||||||
|
suitableNodes.add(n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return suitableNodes
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean anyNode(TaskRun task) {
|
||||||
|
final long reqBytes = getTaskMemoryRequirment(task)
|
||||||
|
final int reqCPUs = getTaskCPURequirement(task)
|
||||||
|
for (WorkerNode n : nodes) {
|
||||||
|
if (n.memoryAmount >= reqBytes && n.CPUCount >= reqCPUs)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,17 +1,17 @@
|
|||||||
package nextflow.k8s.strategies
|
package recreationaltech.plugin.strategies
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import groovy.util.logging.Slf4j
|
import groovy.util.logging.Slf4j
|
||||||
import nextflow.k8s.K8sDVFSClient
|
import recreationaltech.plugin.K8sDVFSClient
|
||||||
import nextflow.k8s.K8sRuntimeEstimator
|
import recreationaltech.plugin.K8sRuntimeEstimator
|
||||||
import nextflow.k8s.K8sSchedulingDecision
|
import recreationaltech.plugin.K8sSchedulingDecision
|
||||||
import nextflow.k8s.K8sSchedulingRequest
|
import recreationaltech.plugin.K8sSchedulingRequest
|
||||||
import nextflow.k8s.K8sSchedulingStrategy
|
import recreationaltech.plugin.K8sSchedulingStrategy
|
||||||
import nextflow.k8s.K8sTaskHandler
|
import recreationaltech.plugin.K8sTaskHandler
|
||||||
import nextflow.k8s.K8sTaskScheduler
|
import recreationaltech.plugin.K8sTaskScheduler
|
||||||
import nextflow.k8s.client.K8sClient
|
import recreationaltech.plugin.client.K8sClient
|
||||||
import nextflow.processor.TaskRun
|
import nextflow.processor.TaskRun
|
||||||
import nextflow.util.ArrayTuple
|
import nextflow.util.Duration
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Implements a scheduling strategy utilizing dvfs to reduce the energy consumption
|
* Implements a scheduling strategy utilizing dvfs to reduce the energy consumption
|
||||||
@@ -132,26 +132,34 @@ class K8sDVFSSchedulingStrategy implements K8sSchedulingStrategy {
|
|||||||
updateFrequency(dvfsClient)
|
updateFrequency(dvfsClient)
|
||||||
}
|
}
|
||||||
|
|
||||||
void taskFinished(TaskRun task, K8sDVFSClient dvfsClient) {
|
// Returns the frequency that was assigned to the task
|
||||||
|
long taskFinished(TaskRun task, K8sDVFSClient dvfsClient) {
|
||||||
|
long f = Long.MAX_VALUE
|
||||||
log.info "[K8s] node ${name}: task ${task.name} finished"
|
log.info "[K8s] node ${name}: task ${task.name} finished"
|
||||||
this.tasks.removeIf {it.task == task}
|
AssignedTask t = this.tasks.find { it.task == task }
|
||||||
updateFrequency(dvfsClient)
|
if (t != null) {
|
||||||
|
f = t.frequency
|
||||||
|
this.tasks.remove(t)
|
||||||
|
updateFrequency(dvfsClient)
|
||||||
|
}
|
||||||
|
return f
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
class SchedulingRequestComparator implements Comparator<K8sSchedulingRequest> {
|
class SchedulingRequestComparator implements Comparator<K8sSchedulingRequest> {
|
||||||
K8sRuntimeEstimator runtimeEstimator
|
K8sRuntimeEstimator runtimeEstimator
|
||||||
long currentTime
|
long currentTime
|
||||||
double avgRuntime
|
double epsilon
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
int compare(K8sSchedulingRequest o1, K8sSchedulingRequest o2) {
|
int compare(K8sSchedulingRequest o1, K8sSchedulingRequest o2) {
|
||||||
// First, check if one of the tasks is (estimated to be) on the critical path
|
// First, check if one of the tasks is (estimated to be) on the critical path
|
||||||
double t1 = runtimeEstimator.estimate(o1.handler)
|
double t1 = runtimeEstimator.estimate(o1.handler)
|
||||||
double t2 = runtimeEstimator.estimate(o2.handler)
|
double t2 = runtimeEstimator.estimate(o2.handler)
|
||||||
if (t1 > avgRuntime && t2 <= avgRuntime)
|
|
||||||
|
if (t1 > t2 + epsilon)
|
||||||
return -1
|
return -1
|
||||||
else if (t1 < avgRuntime && t2 > avgRuntime)
|
else if (t2 > t1 + epsilon)
|
||||||
return 1
|
return 1
|
||||||
|
|
||||||
// Both are not on the critical path. Sort based on the time they spent in the queue
|
// Both are not on the critical path. Sort based on the time they spent in the queue
|
||||||
@@ -170,23 +178,57 @@ class K8sDVFSSchedulingStrategy implements K8sSchedulingStrategy {
|
|||||||
|
|
||||||
private ArrayList<WorkerNode> nodes
|
private ArrayList<WorkerNode> nodes
|
||||||
private HashMap<String, WorkerNode> taskToNode
|
private HashMap<String, WorkerNode> taskToNode
|
||||||
|
|
||||||
private double averageRuntime
|
|
||||||
private long finishedTaskCount
|
|
||||||
|
|
||||||
private long globalMaxFrequency
|
private long globalMaxFrequency
|
||||||
private long globalMinFrequency
|
private long globalMinFrequency
|
||||||
|
|
||||||
private K8sClientGetter clientGetter
|
private K8sClientGetter clientGetter
|
||||||
|
|
||||||
|
private double comparisonEpsilonMillis
|
||||||
|
|
||||||
|
private double[] topRuntimes
|
||||||
|
private double averageRuntime
|
||||||
|
private double finishedTaskCount
|
||||||
|
|
||||||
boolean fullSpeedMode
|
boolean fullSpeedMode
|
||||||
|
|
||||||
K8sDVFSSchedulingStrategy(K8sRuntimeEstimator runtimeEstimator, K8sDVFSClient dvfsClient, K8sClientGetter clientGetter) {
|
K8sDVFSSchedulingStrategy(K8sRuntimeEstimator runtimeEstimator,
|
||||||
|
K8sDVFSClient dvfsClient,
|
||||||
|
K8sClientGetter clientGetter,
|
||||||
|
Duration runtimeComparisonEpsilon,
|
||||||
|
int topRuntimeCount) {
|
||||||
this.runtimeEstimator = runtimeEstimator
|
this.runtimeEstimator = runtimeEstimator
|
||||||
this.dvfsClient = dvfsClient
|
this.dvfsClient = dvfsClient
|
||||||
this.nodes = new ArrayList<>()
|
this.nodes = new ArrayList<>()
|
||||||
this.taskToNode = new HashMap<>();
|
this.taskToNode = new HashMap<>();
|
||||||
this.clientGetter = clientGetter
|
this.clientGetter = clientGetter
|
||||||
|
this.comparisonEpsilonMillis = (double)runtimeComparisonEpsilon.toMillis()
|
||||||
|
this.topRuntimes = new double[topRuntimeCount]
|
||||||
|
for (int i = 0; i < topRuntimeCount; i++) {
|
||||||
|
this.topRuntimes[i] = 0.0
|
||||||
|
}
|
||||||
|
this.averageRuntime = 0.0
|
||||||
|
this.finishedTaskCount = 0.0
|
||||||
|
}
|
||||||
|
|
||||||
|
private boolean isInTopRuntimes(double rt) {
|
||||||
|
for (int i = 0; i < topRuntimes.size(); i++) {
|
||||||
|
if (rt >= topRuntimes[i])
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
private void updateTopRuntimes(double rt) {
|
||||||
|
for (int i = 0; i < topRuntimes.size(); i++) {
|
||||||
|
if (rt > topRuntimes[i]) {
|
||||||
|
/* Move all one down */
|
||||||
|
for (int j = topRuntimes.size() - 1; j > i; j--) {
|
||||||
|
topRuntimes[j] = topRuntimes[j - 1];
|
||||||
|
}
|
||||||
|
topRuntimes[i] = rt
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -199,11 +241,11 @@ class K8sDVFSSchedulingStrategy implements K8sSchedulingStrategy {
|
|||||||
/* Step 1: Sort by task priority. We will attempt to schedule tasks "in order", so that the
|
/* Step 1: Sort by task priority. We will attempt to schedule tasks "in order", so that the
|
||||||
* highest priority tasks are assigned to nodes as soon as possible.
|
* highest priority tasks are assigned to nodes as soon as possible.
|
||||||
*
|
*
|
||||||
* Priority is based on a) an estimation if the task is on the critical path and b) the wait time of the task.
|
* Priority is based on a) the tasks estimated runtime and b) the wait time of the task.
|
||||||
*/
|
*/
|
||||||
SchedulingRequestComparator comparator = new SchedulingRequestComparator()
|
SchedulingRequestComparator comparator = new SchedulingRequestComparator()
|
||||||
comparator.runtimeEstimator = runtimeEstimator
|
comparator.runtimeEstimator = runtimeEstimator
|
||||||
comparator.avgRuntime = averageRuntime
|
comparator.epsilon = comparisonEpsilonMillis
|
||||||
comparator.currentTime = System.currentTimeMillis()
|
comparator.currentTime = System.currentTimeMillis()
|
||||||
queue.sort(comparator)
|
queue.sort(comparator)
|
||||||
|
|
||||||
@@ -214,7 +256,7 @@ class K8sDVFSSchedulingStrategy implements K8sSchedulingStrategy {
|
|||||||
* frequency. If not, we determine a frequency (see below).
|
* frequency. If not, we determine a frequency (see below).
|
||||||
*/
|
*/
|
||||||
final double taskEstimation = runtimeEstimator.estimate(req.handler)
|
final double taskEstimation = runtimeEstimator.estimate(req.handler)
|
||||||
final boolean isCriticalPath = taskEstimation > averageRuntime
|
final boolean isCriticalPath = isInTopRuntimes(taskEstimation)
|
||||||
long frequency = globalMaxFrequency
|
long frequency = globalMaxFrequency
|
||||||
if (!isCriticalPath && !fullSpeedMode) {
|
if (!isCriticalPath && !fullSpeedMode) {
|
||||||
/* Set frequency so that we expect the runtime to be close to the mean runtime. */
|
/* Set frequency so that we expect the runtime to be close to the mean runtime. */
|
||||||
@@ -231,10 +273,10 @@ class K8sDVFSSchedulingStrategy implements K8sSchedulingStrategy {
|
|||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
/* No node can currently execute this task, but it should be possible in the future */
|
/* No node can currently execute this task, but it should be possible in the future */
|
||||||
/* log.info "[K8s] ${req.task} can not be scheduled: ${getTaskMemoryRequirment(req.task)} bytes ${getTaskCPURequirement(req.task)} CPUs"
|
log.info "[K8s] ${req.task} can not be scheduled: ${getTaskMemoryRequirment(req.task)} bytes ${getTaskCPURequirement(req.task)} CPUs"
|
||||||
for (WorkerNode n : this.nodes) {
|
for (WorkerNode n : this.nodes) {
|
||||||
log.info "[K8s] node ${n.name} - ${n.availableMemory}, ${n.availableCPUs}"
|
log.info "[K8s] node ${n.name} - ${n.availableMemory}, ${n.availableCPUs}"
|
||||||
} */
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -273,20 +315,11 @@ class K8sDVFSSchedulingStrategy implements K8sSchedulingStrategy {
|
|||||||
|
|
||||||
@Override
|
@Override
|
||||||
synchronized void taskFinished(K8sTaskHandler task) {
|
synchronized void taskFinished(K8sTaskHandler task) {
|
||||||
/* TODO: This just uses elapsed wall-clock time, regardless of the frequency used to execute the task.
|
long freq = globalMaxFrequency
|
||||||
* This will skew the average towards longer runtimes, which is undesirable, because it will lead to more
|
|
||||||
* tasks classified as "critical path".
|
|
||||||
* A simple (rough) solution could be to keep track of the tasks "relative" frequency and just scale the
|
|
||||||
* elapsed time based on that.
|
|
||||||
*/
|
|
||||||
double runtime = (double)(task.getCompleteTimeMillis() - task.getStartTimeMillis())
|
|
||||||
averageRuntime = (runtime + finishedTaskCount * averageRuntime) / (finishedTaskCount + 1)
|
|
||||||
finishedTaskCount += 1
|
|
||||||
|
|
||||||
/* Free resources allocated by this task */
|
/* Free resources allocated by this task */
|
||||||
WorkerNode node = taskToNode.get(task.task.hash.toString())
|
WorkerNode node = taskToNode.get(task.task.hash.toString())
|
||||||
if (node != null) {
|
if (node != null) {
|
||||||
node.taskFinished(task.task, dvfsClient)
|
freq = node.taskFinished(task.task, dvfsClient)
|
||||||
taskToNode.remove(task.task.hash.toString())
|
taskToNode.remove(task.task.hash.toString())
|
||||||
} else {
|
} else {
|
||||||
log.warn "[K8s] no node recorded for task ${task.toString()}"
|
log.warn "[K8s] no node recorded for task ${task.toString()}"
|
||||||
@@ -294,8 +327,17 @@ class K8sDVFSSchedulingStrategy implements K8sSchedulingStrategy {
|
|||||||
|
|
||||||
log.info "[K8s] task ${task.toString()} finished - ${taskToNode.size()} tasks running"
|
log.info "[K8s] task ${task.toString()} finished - ${taskToNode.size()} tasks running"
|
||||||
if (node != null) {
|
if (node != null) {
|
||||||
log.info "[K8s] task ran on node ${node.name} - ${node.availableMemory} bytes ${node.availableCPUs}"
|
log.info "[K8s] task ran on node ${node.name} - ${node.availableMemory} bytes ${node.availableCPUs} at ${freq}/${globalMaxFrequency} Hz (${(double)freq / (double)globalMaxFrequency}%)"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* We scale the runtime by the tasks relative frequency to avoid skewing the average runtime towards
|
||||||
|
* longer runtimes. This is obviously only a rough approximation.
|
||||||
|
*/
|
||||||
|
double runtime = (double)(task.getCompleteTimeMillis() - task.getStartTimeMillis())
|
||||||
|
runtime *= (double)freq / (double)globalMaxFrequency
|
||||||
|
averageRuntime = (runtime + finishedTaskCount * averageRuntime) / (finishedTaskCount + 1.0)
|
||||||
|
finishedTaskCount += 1.0
|
||||||
|
updateTopRuntimes(runtime)
|
||||||
}
|
}
|
||||||
|
|
||||||
private synchronized boolean initNodes(K8sTaskScheduler scheduler) {
|
private synchronized boolean initNodes(K8sTaskScheduler scheduler) {
|
||||||
@@ -1,11 +1,11 @@
|
|||||||
package nextflow.k8s.strategies
|
package recreationaltech.plugin.strategies
|
||||||
|
|
||||||
import groovy.transform.CompileStatic
|
import groovy.transform.CompileStatic
|
||||||
import nextflow.k8s.K8sSchedulingDecision
|
import recreationaltech.plugin.K8sSchedulingDecision
|
||||||
import nextflow.k8s.K8sSchedulingRequest
|
import recreationaltech.plugin.K8sSchedulingRequest
|
||||||
import nextflow.k8s.K8sSchedulingStrategy
|
import recreationaltech.plugin.K8sSchedulingStrategy
|
||||||
import nextflow.k8s.K8sTaskHandler
|
import recreationaltech.plugin.K8sTaskHandler
|
||||||
import nextflow.k8s.K8sTaskScheduler
|
import recreationaltech.plugin.K8sTaskScheduler
|
||||||
|
|
||||||
@CompileStatic
|
@CompileStatic
|
||||||
class K8sHashSchedulingStrategy implements K8sSchedulingStrategy {
|
class K8sHashSchedulingStrategy implements K8sSchedulingStrategy {
|
||||||
@@ -14,14 +14,14 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import nextflow.BuildInfo
|
import nextflow.BuildInfo
|
||||||
import nextflow.SysEnv
|
import nextflow.SysEnv
|
||||||
import nextflow.k8s.client.ClientConfig
|
import recreationaltech.plugin.client.ClientConfig
|
||||||
import nextflow.k8s.model.PodEnv
|
import recreationaltech.plugin.model.PodEnv
|
||||||
import nextflow.k8s.model.PodSecurityContext
|
import recreationaltech.plugin.model.PodSecurityContext
|
||||||
import nextflow.k8s.model.PodVolumeClaim
|
import recreationaltech.plugin.model.PodVolumeClaim
|
||||||
import nextflow.util.Duration
|
import nextflow.util.Duration
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
import spock.lang.Unroll
|
import spock.lang.Unroll
|
||||||
@@ -14,19 +14,19 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import java.nio.file.Files
|
import java.nio.file.Files
|
||||||
|
|
||||||
import nextflow.cli.CliOptions
|
import nextflow.cli.CliOptions
|
||||||
import nextflow.cli.CmdKubeRun
|
import nextflow.cli.CmdKubeRun
|
||||||
import nextflow.cli.Launcher
|
import nextflow.cli.Launcher
|
||||||
import nextflow.k8s.client.ClientConfig
|
import recreationaltech.plugin.client.ClientConfig
|
||||||
import nextflow.k8s.client.K8sClient
|
import recreationaltech.plugin.client.K8sClient
|
||||||
import nextflow.k8s.model.PodMountConfig
|
import recreationaltech.plugin.model.PodMountConfig
|
||||||
import nextflow.k8s.model.PodOptions
|
import recreationaltech.plugin.model.PodOptions
|
||||||
import nextflow.k8s.model.PodSpecBuilder
|
import recreationaltech.plugin.model.PodSpecBuilder
|
||||||
import nextflow.k8s.model.PodVolumeClaim
|
import recreationaltech.plugin.model.PodVolumeClaim
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
import spock.lang.Unroll
|
import spock.lang.Unroll
|
||||||
/**
|
/**
|
||||||
@@ -14,13 +14,13 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import java.util.concurrent.TimeUnit
|
import java.util.concurrent.TimeUnit
|
||||||
|
|
||||||
import com.google.common.cache.CacheBuilder
|
import com.google.common.cache.CacheBuilder
|
||||||
import nextflow.k8s.client.ClientConfig
|
import recreationaltech.plugin.client.ClientConfig
|
||||||
import nextflow.k8s.client.K8sClient
|
import recreationaltech.plugin.client.K8sClient
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import nextflow.k8s.client.K8sClient
|
import recreationaltech.plugin.client.K8sClient
|
||||||
import nextflow.k8s.model.PodSpecBuilder
|
import recreationaltech.plugin.model.PodSpecBuilder
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
|
|
||||||
class K8sNodeInitDeployerTest extends Specification {
|
class K8sNodeInitDeployerTest extends Specification {
|
||||||
@@ -52,6 +52,7 @@ class K8sNodeInitDeployerTest extends Specification {
|
|||||||
spec.metadata.namespace == 'default'
|
spec.metadata.namespace == 'default'
|
||||||
spec.spec.nodeName == 'node-a'
|
spec.spec.nodeName == 'node-a'
|
||||||
spec.spec.restartPolicy == 'Never'
|
spec.spec.restartPolicy == 'Never'
|
||||||
|
spec.spec.hostNetwork == true
|
||||||
|
|
||||||
def container = spec.spec.containers[0]
|
def container = spec.spec.containers[0]
|
||||||
container.name == 'nf-init-node-a'
|
container.name == 'nf-init-node-a'
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import nextflow.processor.TaskRun
|
import nextflow.processor.TaskRun
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import java.nio.file.Files
|
import java.nio.file.Files
|
||||||
import java.nio.file.Path
|
import java.nio.file.Path
|
||||||
@@ -26,18 +26,18 @@ import nextflow.exception.NodeTerminationException
|
|||||||
import nextflow.file.http.XPath
|
import nextflow.file.http.XPath
|
||||||
import nextflow.fusion.FusionConfig
|
import nextflow.fusion.FusionConfig
|
||||||
import nextflow.fusion.FusionScriptLauncher
|
import nextflow.fusion.FusionScriptLauncher
|
||||||
import nextflow.k8s.client.ClientConfig
|
import recreationaltech.plugin.client.ClientConfig
|
||||||
import nextflow.k8s.client.K8sClient
|
import recreationaltech.plugin.client.K8sClient
|
||||||
import nextflow.k8s.client.K8sResponseException
|
import recreationaltech.plugin.client.K8sResponseException
|
||||||
import nextflow.k8s.client.K8sResponseJson
|
import recreationaltech.plugin.client.K8sResponseJson
|
||||||
import nextflow.k8s.client.PodUnschedulableException
|
import recreationaltech.plugin.client.PodUnschedulableException
|
||||||
import nextflow.k8s.model.PodEnv
|
import recreationaltech.plugin.model.PodEnv
|
||||||
import nextflow.k8s.model.PodHostMount
|
import recreationaltech.plugin.model.PodHostMount
|
||||||
import nextflow.k8s.model.PodMountConfig
|
import recreationaltech.plugin.model.PodMountConfig
|
||||||
import nextflow.k8s.model.PodMountSecret
|
import recreationaltech.plugin.model.PodMountSecret
|
||||||
import nextflow.k8s.model.PodOptions
|
import recreationaltech.plugin.model.PodOptions
|
||||||
import nextflow.k8s.model.PodSpecBuilder
|
import recreationaltech.plugin.model.PodSpecBuilder
|
||||||
import nextflow.k8s.model.PodVolumeClaim
|
import recreationaltech.plugin.model.PodVolumeClaim
|
||||||
import nextflow.processor.TaskBean
|
import nextflow.processor.TaskBean
|
||||||
import nextflow.processor.TaskConfig
|
import nextflow.processor.TaskConfig
|
||||||
import nextflow.processor.TaskProcessor
|
import nextflow.processor.TaskProcessor
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s
|
package recreationaltech.plugin
|
||||||
|
|
||||||
import java.nio.file.Files
|
import java.nio.file.Files
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import java.nio.file.Files
|
import java.nio.file.Files
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
import javax.net.ssl.KeyManager
|
import javax.net.ssl.KeyManager
|
||||||
import java.nio.file.Files
|
import java.nio.file.Files
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import nextflow.exception.K8sOutOfCpuException
|
import nextflow.exception.K8sOutOfCpuException
|
||||||
import nextflow.exception.K8sOutOfMemoryException
|
import nextflow.exception.K8sOutOfMemoryException
|
||||||
@@ -1139,6 +1139,61 @@ class K8sClientTest extends Specification {
|
|||||||
result.metadata.name == 'test-node'
|
result.metadata.name == 'test-node'
|
||||||
}
|
}
|
||||||
|
|
||||||
|
def 'should get node external IP with ExternalIP address' () {
|
||||||
|
given:
|
||||||
|
def JSON = '''
|
||||||
|
{
|
||||||
|
"kind": "Node",
|
||||||
|
"apiVersion": "v1",
|
||||||
|
"metadata": {
|
||||||
|
"name": "test-node"
|
||||||
|
},
|
||||||
|
"status": {
|
||||||
|
"addresses": [
|
||||||
|
{"type": "InternalIP", "address": "192.168.1.100"},
|
||||||
|
{"type": "ExternalIP", "address": "203.0.113.10"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
'''
|
||||||
|
|
||||||
|
def client = Spy(K8sClient)
|
||||||
|
final NODE_NAME = 'test-node'
|
||||||
|
|
||||||
|
when:
|
||||||
|
def result = client.getNodeExternalIp(NODE_NAME)
|
||||||
|
then:
|
||||||
|
1 * client.nodeDescribe(NODE_NAME) >> new K8sResponseJson(JSON)
|
||||||
|
result == '203.0.113.10'
|
||||||
|
}
|
||||||
|
|
||||||
|
def 'should get node external IP fallback to InternalIP' () {
|
||||||
|
given:
|
||||||
|
def JSON = '''
|
||||||
|
{
|
||||||
|
"kind": "Node",
|
||||||
|
"apiVersion": "v1",
|
||||||
|
"metadata": {
|
||||||
|
"name": "test-node"
|
||||||
|
},
|
||||||
|
"status": {
|
||||||
|
"addresses": [
|
||||||
|
{"type": "InternalIP", "address": "192.168.1.100"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
'''
|
||||||
|
|
||||||
|
def client = Spy(K8sClient)
|
||||||
|
final NODE_NAME = 'test-node'
|
||||||
|
|
||||||
|
when:
|
||||||
|
def result = client.getNodeExternalIp(NODE_NAME)
|
||||||
|
then:
|
||||||
|
1 * client.nodeDescribe(NODE_NAME) >> new K8sResponseJson(JSON)
|
||||||
|
result == '192.168.1.100'
|
||||||
|
}
|
||||||
|
|
||||||
def 'should get node CPU capacity' () {
|
def 'should get node CPU capacity' () {
|
||||||
given:
|
given:
|
||||||
def JSON = '''
|
def JSON = '''
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.client
|
package recreationaltech.plugin.client
|
||||||
|
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
|
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import nextflow.executor.res.AcceleratorResource
|
import nextflow.executor.res.AcceleratorResource
|
||||||
import nextflow.util.MemoryUnit
|
import nextflow.util.MemoryUnit
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
* limitations under the License.
|
* limitations under the License.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
package nextflow.k8s.model
|
package recreationaltech.plugin.model
|
||||||
|
|
||||||
import spock.lang.Specification
|
import spock.lang.Specification
|
||||||
/**
|
/**
|
||||||
@@ -1,184 +0,0 @@
|
|||||||
---
|
|
||||||
description: Perform a non-destructive cross-artifact consistency and quality analysis across spec.md, plan.md, and tasks.md after task generation.
|
|
||||||
---
|
|
||||||
|
|
||||||
## User Input
|
|
||||||
|
|
||||||
```text
|
|
||||||
$ARGUMENTS
|
|
||||||
```
|
|
||||||
|
|
||||||
You **MUST** consider the user input before proceeding (if not empty).
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
|
|
||||||
Identify inconsistencies, duplications, ambiguities, and underspecified items across the three core artifacts (`spec.md`, `plan.md`, `tasks.md`) before implementation. This command MUST run only after `/speckit.tasks` has successfully produced a complete `tasks.md`.
|
|
||||||
|
|
||||||
## Operating Constraints
|
|
||||||
|
|
||||||
**STRICTLY READ-ONLY**: Do **not** modify any files. Output a structured analysis report. Offer an optional remediation plan (user must explicitly approve before any follow-up editing commands would be invoked manually).
|
|
||||||
|
|
||||||
**Constitution Authority**: The project constitution (`.specify/memory/constitution.md`) is **non-negotiable** within this analysis scope. Constitution conflicts are automatically CRITICAL and require adjustment of the spec, plan, or tasks—not dilution, reinterpretation, or silent ignoring of the principle. If a principle itself needs to change, that must occur in a separate, explicit constitution update outside `/speckit.analyze`.
|
|
||||||
|
|
||||||
## Execution Steps
|
|
||||||
|
|
||||||
### 1. Initialize Analysis Context
|
|
||||||
|
|
||||||
Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks` once from repo root and parse JSON for FEATURE_DIR and AVAILABLE_DOCS. Derive absolute paths:
|
|
||||||
|
|
||||||
- SPEC = FEATURE_DIR/spec.md
|
|
||||||
- PLAN = FEATURE_DIR/plan.md
|
|
||||||
- TASKS = FEATURE_DIR/tasks.md
|
|
||||||
|
|
||||||
Abort with an error message if any required file is missing (instruct the user to run missing prerequisite command).
|
|
||||||
For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
|
||||||
|
|
||||||
### 2. Load Artifacts (Progressive Disclosure)
|
|
||||||
|
|
||||||
Load only the minimal necessary context from each artifact:
|
|
||||||
|
|
||||||
**From spec.md:**
|
|
||||||
|
|
||||||
- Overview/Context
|
|
||||||
- Functional Requirements
|
|
||||||
- Non-Functional Requirements
|
|
||||||
- User Stories
|
|
||||||
- Edge Cases (if present)
|
|
||||||
|
|
||||||
**From plan.md:**
|
|
||||||
|
|
||||||
- Architecture/stack choices
|
|
||||||
- Data Model references
|
|
||||||
- Phases
|
|
||||||
- Technical constraints
|
|
||||||
|
|
||||||
**From tasks.md:**
|
|
||||||
|
|
||||||
- Task IDs
|
|
||||||
- Descriptions
|
|
||||||
- Phase grouping
|
|
||||||
- Parallel markers [P]
|
|
||||||
- Referenced file paths
|
|
||||||
|
|
||||||
**From constitution:**
|
|
||||||
|
|
||||||
- Load `.specify/memory/constitution.md` for principle validation
|
|
||||||
|
|
||||||
### 3. Build Semantic Models
|
|
||||||
|
|
||||||
Create internal representations (do not include raw artifacts in output):
|
|
||||||
|
|
||||||
- **Requirements inventory**: Each functional + non-functional requirement with a stable key (derive slug based on imperative phrase; e.g., "User can upload file" → `user-can-upload-file`)
|
|
||||||
- **User story/action inventory**: Discrete user actions with acceptance criteria
|
|
||||||
- **Task coverage mapping**: Map each task to one or more requirements or stories (inference by keyword / explicit reference patterns like IDs or key phrases)
|
|
||||||
- **Constitution rule set**: Extract principle names and MUST/SHOULD normative statements
|
|
||||||
|
|
||||||
### 4. Detection Passes (Token-Efficient Analysis)
|
|
||||||
|
|
||||||
Focus on high-signal findings. Limit to 50 findings total; aggregate remainder in overflow summary.
|
|
||||||
|
|
||||||
#### A. Duplication Detection
|
|
||||||
|
|
||||||
- Identify near-duplicate requirements
|
|
||||||
- Mark lower-quality phrasing for consolidation
|
|
||||||
|
|
||||||
#### B. Ambiguity Detection
|
|
||||||
|
|
||||||
- Flag vague adjectives (fast, scalable, secure, intuitive, robust) lacking measurable criteria
|
|
||||||
- Flag unresolved placeholders (TODO, TKTK, ???, `<placeholder>`, etc.)
|
|
||||||
|
|
||||||
#### C. Underspecification
|
|
||||||
|
|
||||||
- Requirements with verbs but missing object or measurable outcome
|
|
||||||
- User stories missing acceptance criteria alignment
|
|
||||||
- Tasks referencing files or components not defined in spec/plan
|
|
||||||
|
|
||||||
#### D. Constitution Alignment
|
|
||||||
|
|
||||||
- Any requirement or plan element conflicting with a MUST principle
|
|
||||||
- Missing mandated sections or quality gates from constitution
|
|
||||||
|
|
||||||
#### E. Coverage Gaps
|
|
||||||
|
|
||||||
- Requirements with zero associated tasks
|
|
||||||
- Tasks with no mapped requirement/story
|
|
||||||
- Non-functional requirements not reflected in tasks (e.g., performance, security)
|
|
||||||
|
|
||||||
#### F. Inconsistency
|
|
||||||
|
|
||||||
- Terminology drift (same concept named differently across files)
|
|
||||||
- Data entities referenced in plan but absent in spec (or vice versa)
|
|
||||||
- Task ordering contradictions (e.g., integration tasks before foundational setup tasks without dependency note)
|
|
||||||
- Conflicting requirements (e.g., one requires Next.js while other specifies Vue)
|
|
||||||
|
|
||||||
### 5. Severity Assignment
|
|
||||||
|
|
||||||
Use this heuristic to prioritize findings:
|
|
||||||
|
|
||||||
- **CRITICAL**: Violates constitution MUST, missing core spec artifact, or requirement with zero coverage that blocks baseline functionality
|
|
||||||
- **HIGH**: Duplicate or conflicting requirement, ambiguous security/performance attribute, untestable acceptance criterion
|
|
||||||
- **MEDIUM**: Terminology drift, missing non-functional task coverage, underspecified edge case
|
|
||||||
- **LOW**: Style/wording improvements, minor redundancy not affecting execution order
|
|
||||||
|
|
||||||
### 6. Produce Compact Analysis Report
|
|
||||||
|
|
||||||
Output a Markdown report (no file writes) with the following structure:
|
|
||||||
|
|
||||||
## Specification Analysis Report
|
|
||||||
|
|
||||||
| ID | Category | Severity | Location(s) | Summary | Recommendation |
|
|
||||||
|----|----------|----------|-------------|---------|----------------|
|
|
||||||
| A1 | Duplication | HIGH | spec.md:L120-134 | Two similar requirements ... | Merge phrasing; keep clearer version |
|
|
||||||
|
|
||||||
(Add one row per finding; generate stable IDs prefixed by category initial.)
|
|
||||||
|
|
||||||
**Coverage Summary Table:**
|
|
||||||
|
|
||||||
| Requirement Key | Has Task? | Task IDs | Notes |
|
|
||||||
|-----------------|-----------|----------|-------|
|
|
||||||
|
|
||||||
**Constitution Alignment Issues:** (if any)
|
|
||||||
|
|
||||||
**Unmapped Tasks:** (if any)
|
|
||||||
|
|
||||||
**Metrics:**
|
|
||||||
|
|
||||||
- Total Requirements
|
|
||||||
- Total Tasks
|
|
||||||
- Coverage % (requirements with >=1 task)
|
|
||||||
- Ambiguity Count
|
|
||||||
- Duplication Count
|
|
||||||
- Critical Issues Count
|
|
||||||
|
|
||||||
### 7. Provide Next Actions
|
|
||||||
|
|
||||||
At end of report, output a concise Next Actions block:
|
|
||||||
|
|
||||||
- If CRITICAL issues exist: Recommend resolving before `/speckit.implement`
|
|
||||||
- If only LOW/MEDIUM: User may proceed, but provide improvement suggestions
|
|
||||||
- Provide explicit command suggestions: e.g., "Run /speckit.specify with refinement", "Run /speckit.plan to adjust architecture", "Manually edit tasks.md to add coverage for 'performance-metrics'"
|
|
||||||
|
|
||||||
### 8. Offer Remediation
|
|
||||||
|
|
||||||
Ask the user: "Would you like me to suggest concrete remediation edits for the top N issues?" (Do NOT apply them automatically.)
|
|
||||||
|
|
||||||
## Operating Principles
|
|
||||||
|
|
||||||
### Context Efficiency
|
|
||||||
|
|
||||||
- **Minimal high-signal tokens**: Focus on actionable findings, not exhaustive documentation
|
|
||||||
- **Progressive disclosure**: Load artifacts incrementally; don't dump all content into analysis
|
|
||||||
- **Token-efficient output**: Limit findings table to 50 rows; summarize overflow
|
|
||||||
- **Deterministic results**: Rerunning without changes should produce consistent IDs and counts
|
|
||||||
|
|
||||||
### Analysis Guidelines
|
|
||||||
|
|
||||||
- **NEVER modify files** (this is read-only analysis)
|
|
||||||
- **NEVER hallucinate missing sections** (if absent, report them accurately)
|
|
||||||
- **Prioritize constitution violations** (these are always CRITICAL)
|
|
||||||
- **Use examples over exhaustive rules** (cite specific instances, not generic patterns)
|
|
||||||
- **Report zero issues gracefully** (emit success report with coverage statistics)
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
$ARGUMENTS
|
|
||||||
@@ -1,294 +0,0 @@
|
|||||||
---
|
|
||||||
description: Generate a custom checklist for the current feature based on user requirements.
|
|
||||||
---
|
|
||||||
|
|
||||||
## Checklist Purpose: "Unit Tests for English"
|
|
||||||
|
|
||||||
**CRITICAL CONCEPT**: Checklists are **UNIT TESTS FOR REQUIREMENTS WRITING** - they validate the quality, clarity, and completeness of requirements in a given domain.
|
|
||||||
|
|
||||||
**NOT for verification/testing**:
|
|
||||||
|
|
||||||
- ❌ NOT "Verify the button clicks correctly"
|
|
||||||
- ❌ NOT "Test error handling works"
|
|
||||||
- ❌ NOT "Confirm the API returns 200"
|
|
||||||
- ❌ NOT checking if code/implementation matches the spec
|
|
||||||
|
|
||||||
**FOR requirements quality validation**:
|
|
||||||
|
|
||||||
- ✅ "Are visual hierarchy requirements defined for all card types?" (completeness)
|
|
||||||
- ✅ "Is 'prominent display' quantified with specific sizing/positioning?" (clarity)
|
|
||||||
- ✅ "Are hover state requirements consistent across all interactive elements?" (consistency)
|
|
||||||
- ✅ "Are accessibility requirements defined for keyboard navigation?" (coverage)
|
|
||||||
- ✅ "Does the spec define what happens when logo image fails to load?" (edge cases)
|
|
||||||
|
|
||||||
**Metaphor**: If your spec is code written in English, the checklist is its unit test suite. You're testing whether the requirements are well-written, complete, unambiguous, and ready for implementation - NOT whether the implementation works.
|
|
||||||
|
|
||||||
## User Input
|
|
||||||
|
|
||||||
```text
|
|
||||||
$ARGUMENTS
|
|
||||||
```
|
|
||||||
|
|
||||||
You **MUST** consider the user input before proceeding (if not empty).
|
|
||||||
|
|
||||||
## Execution Steps
|
|
||||||
|
|
||||||
1. **Setup**: Run `.specify/scripts/bash/check-prerequisites.sh --json` from repo root and parse JSON for FEATURE_DIR and AVAILABLE_DOCS list.
|
|
||||||
- All file paths must be absolute.
|
|
||||||
- For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
|
||||||
|
|
||||||
2. **Clarify intent (dynamic)**: Derive up to THREE initial contextual clarifying questions (no pre-baked catalog). They MUST:
|
|
||||||
- Be generated from the user's phrasing + extracted signals from spec/plan/tasks
|
|
||||||
- Only ask about information that materially changes checklist content
|
|
||||||
- Be skipped individually if already unambiguous in `$ARGUMENTS`
|
|
||||||
- Prefer precision over breadth
|
|
||||||
|
|
||||||
Generation algorithm:
|
|
||||||
1. Extract signals: feature domain keywords (e.g., auth, latency, UX, API), risk indicators ("critical", "must", "compliance"), stakeholder hints ("QA", "review", "security team"), and explicit deliverables ("a11y", "rollback", "contracts").
|
|
||||||
2. Cluster signals into candidate focus areas (max 4) ranked by relevance.
|
|
||||||
3. Identify probable audience & timing (author, reviewer, QA, release) if not explicit.
|
|
||||||
4. Detect missing dimensions: scope breadth, depth/rigor, risk emphasis, exclusion boundaries, measurable acceptance criteria.
|
|
||||||
5. Formulate questions chosen from these archetypes:
|
|
||||||
- Scope refinement (e.g., "Should this include integration touchpoints with X and Y or stay limited to local module correctness?")
|
|
||||||
- Risk prioritization (e.g., "Which of these potential risk areas should receive mandatory gating checks?")
|
|
||||||
- Depth calibration (e.g., "Is this a lightweight pre-commit sanity list or a formal release gate?")
|
|
||||||
- Audience framing (e.g., "Will this be used by the author only or peers during PR review?")
|
|
||||||
- Boundary exclusion (e.g., "Should we explicitly exclude performance tuning items this round?")
|
|
||||||
- Scenario class gap (e.g., "No recovery flows detected—are rollback / partial failure paths in scope?")
|
|
||||||
|
|
||||||
Question formatting rules:
|
|
||||||
- If presenting options, generate a compact table with columns: Option | Candidate | Why It Matters
|
|
||||||
- Limit to A–E options maximum; omit table if a free-form answer is clearer
|
|
||||||
- Never ask the user to restate what they already said
|
|
||||||
- Avoid speculative categories (no hallucination). If uncertain, ask explicitly: "Confirm whether X belongs in scope."
|
|
||||||
|
|
||||||
Defaults when interaction impossible:
|
|
||||||
- Depth: Standard
|
|
||||||
- Audience: Reviewer (PR) if code-related; Author otherwise
|
|
||||||
- Focus: Top 2 relevance clusters
|
|
||||||
|
|
||||||
Output the questions (label Q1/Q2/Q3). After answers: if ≥2 scenario classes (Alternate / Exception / Recovery / Non-Functional domain) remain unclear, you MAY ask up to TWO more targeted follow‑ups (Q4/Q5) with a one-line justification each (e.g., "Unresolved recovery path risk"). Do not exceed five total questions. Skip escalation if user explicitly declines more.
|
|
||||||
|
|
||||||
3. **Understand user request**: Combine `$ARGUMENTS` + clarifying answers:
|
|
||||||
- Derive checklist theme (e.g., security, review, deploy, ux)
|
|
||||||
- Consolidate explicit must-have items mentioned by user
|
|
||||||
- Map focus selections to category scaffolding
|
|
||||||
- Infer any missing context from spec/plan/tasks (do NOT hallucinate)
|
|
||||||
|
|
||||||
4. **Load feature context**: Read from FEATURE_DIR:
|
|
||||||
- spec.md: Feature requirements and scope
|
|
||||||
- plan.md (if exists): Technical details, dependencies
|
|
||||||
- tasks.md (if exists): Implementation tasks
|
|
||||||
|
|
||||||
**Context Loading Strategy**:
|
|
||||||
- Load only necessary portions relevant to active focus areas (avoid full-file dumping)
|
|
||||||
- Prefer summarizing long sections into concise scenario/requirement bullets
|
|
||||||
- Use progressive disclosure: add follow-on retrieval only if gaps detected
|
|
||||||
- If source docs are large, generate interim summary items instead of embedding raw text
|
|
||||||
|
|
||||||
5. **Generate checklist** - Create "Unit Tests for Requirements":
|
|
||||||
- Create `FEATURE_DIR/checklists/` directory if it doesn't exist
|
|
||||||
- Generate unique checklist filename:
|
|
||||||
- Use short, descriptive name based on domain (e.g., `ux.md`, `api.md`, `security.md`)
|
|
||||||
- Format: `[domain].md`
|
|
||||||
- If file exists, append to existing file
|
|
||||||
- Number items sequentially starting from CHK001
|
|
||||||
- Each `/speckit.checklist` run creates a NEW file (never overwrites existing checklists)
|
|
||||||
|
|
||||||
**CORE PRINCIPLE - Test the Requirements, Not the Implementation**:
|
|
||||||
Every checklist item MUST evaluate the REQUIREMENTS THEMSELVES for:
|
|
||||||
- **Completeness**: Are all necessary requirements present?
|
|
||||||
- **Clarity**: Are requirements unambiguous and specific?
|
|
||||||
- **Consistency**: Do requirements align with each other?
|
|
||||||
- **Measurability**: Can requirements be objectively verified?
|
|
||||||
- **Coverage**: Are all scenarios/edge cases addressed?
|
|
||||||
|
|
||||||
**Category Structure** - Group items by requirement quality dimensions:
|
|
||||||
- **Requirement Completeness** (Are all necessary requirements documented?)
|
|
||||||
- **Requirement Clarity** (Are requirements specific and unambiguous?)
|
|
||||||
- **Requirement Consistency** (Do requirements align without conflicts?)
|
|
||||||
- **Acceptance Criteria Quality** (Are success criteria measurable?)
|
|
||||||
- **Scenario Coverage** (Are all flows/cases addressed?)
|
|
||||||
- **Edge Case Coverage** (Are boundary conditions defined?)
|
|
||||||
- **Non-Functional Requirements** (Performance, Security, Accessibility, etc. - are they specified?)
|
|
||||||
- **Dependencies & Assumptions** (Are they documented and validated?)
|
|
||||||
- **Ambiguities & Conflicts** (What needs clarification?)
|
|
||||||
|
|
||||||
**HOW TO WRITE CHECKLIST ITEMS - "Unit Tests for English"**:
|
|
||||||
|
|
||||||
❌ **WRONG** (Testing implementation):
|
|
||||||
- "Verify landing page displays 3 episode cards"
|
|
||||||
- "Test hover states work on desktop"
|
|
||||||
- "Confirm logo click navigates home"
|
|
||||||
|
|
||||||
✅ **CORRECT** (Testing requirements quality):
|
|
||||||
- "Are the exact number and layout of featured episodes specified?" [Completeness]
|
|
||||||
- "Is 'prominent display' quantified with specific sizing/positioning?" [Clarity]
|
|
||||||
- "Are hover state requirements consistent across all interactive elements?" [Consistency]
|
|
||||||
- "Are keyboard navigation requirements defined for all interactive UI?" [Coverage]
|
|
||||||
- "Is the fallback behavior specified when logo image fails to load?" [Edge Cases]
|
|
||||||
- "Are loading states defined for asynchronous episode data?" [Completeness]
|
|
||||||
- "Does the spec define visual hierarchy for competing UI elements?" [Clarity]
|
|
||||||
|
|
||||||
**ITEM STRUCTURE**:
|
|
||||||
Each item should follow this pattern:
|
|
||||||
- Question format asking about requirement quality
|
|
||||||
- Focus on what's WRITTEN (or not written) in the spec/plan
|
|
||||||
- Include quality dimension in brackets [Completeness/Clarity/Consistency/etc.]
|
|
||||||
- Reference spec section `[Spec §X.Y]` when checking existing requirements
|
|
||||||
- Use `[Gap]` marker when checking for missing requirements
|
|
||||||
|
|
||||||
**EXAMPLES BY QUALITY DIMENSION**:
|
|
||||||
|
|
||||||
Completeness:
|
|
||||||
- "Are error handling requirements defined for all API failure modes? [Gap]"
|
|
||||||
- "Are accessibility requirements specified for all interactive elements? [Completeness]"
|
|
||||||
- "Are mobile breakpoint requirements defined for responsive layouts? [Gap]"
|
|
||||||
|
|
||||||
Clarity:
|
|
||||||
- "Is 'fast loading' quantified with specific timing thresholds? [Clarity, Spec §NFR-2]"
|
|
||||||
- "Are 'related episodes' selection criteria explicitly defined? [Clarity, Spec §FR-5]"
|
|
||||||
- "Is 'prominent' defined with measurable visual properties? [Ambiguity, Spec §FR-4]"
|
|
||||||
|
|
||||||
Consistency:
|
|
||||||
- "Do navigation requirements align across all pages? [Consistency, Spec §FR-10]"
|
|
||||||
- "Are card component requirements consistent between landing and detail pages? [Consistency]"
|
|
||||||
|
|
||||||
Coverage:
|
|
||||||
- "Are requirements defined for zero-state scenarios (no episodes)? [Coverage, Edge Case]"
|
|
||||||
- "Are concurrent user interaction scenarios addressed? [Coverage, Gap]"
|
|
||||||
- "Are requirements specified for partial data loading failures? [Coverage, Exception Flow]"
|
|
||||||
|
|
||||||
Measurability:
|
|
||||||
- "Are visual hierarchy requirements measurable/testable? [Acceptance Criteria, Spec §FR-1]"
|
|
||||||
- "Can 'balanced visual weight' be objectively verified? [Measurability, Spec §FR-2]"
|
|
||||||
|
|
||||||
**Scenario Classification & Coverage** (Requirements Quality Focus):
|
|
||||||
- Check if requirements exist for: Primary, Alternate, Exception/Error, Recovery, Non-Functional scenarios
|
|
||||||
- For each scenario class, ask: "Are [scenario type] requirements complete, clear, and consistent?"
|
|
||||||
- If scenario class missing: "Are [scenario type] requirements intentionally excluded or missing? [Gap]"
|
|
||||||
- Include resilience/rollback when state mutation occurs: "Are rollback requirements defined for migration failures? [Gap]"
|
|
||||||
|
|
||||||
**Traceability Requirements**:
|
|
||||||
- MINIMUM: ≥80% of items MUST include at least one traceability reference
|
|
||||||
- Each item should reference: spec section `[Spec §X.Y]`, or use markers: `[Gap]`, `[Ambiguity]`, `[Conflict]`, `[Assumption]`
|
|
||||||
- If no ID system exists: "Is a requirement & acceptance criteria ID scheme established? [Traceability]"
|
|
||||||
|
|
||||||
**Surface & Resolve Issues** (Requirements Quality Problems):
|
|
||||||
Ask questions about the requirements themselves:
|
|
||||||
- Ambiguities: "Is the term 'fast' quantified with specific metrics? [Ambiguity, Spec §NFR-1]"
|
|
||||||
- Conflicts: "Do navigation requirements conflict between §FR-10 and §FR-10a? [Conflict]"
|
|
||||||
- Assumptions: "Is the assumption of 'always available podcast API' validated? [Assumption]"
|
|
||||||
- Dependencies: "Are external podcast API requirements documented? [Dependency, Gap]"
|
|
||||||
- Missing definitions: "Is 'visual hierarchy' defined with measurable criteria? [Gap]"
|
|
||||||
|
|
||||||
**Content Consolidation**:
|
|
||||||
- Soft cap: If raw candidate items > 40, prioritize by risk/impact
|
|
||||||
- Merge near-duplicates checking the same requirement aspect
|
|
||||||
- If >5 low-impact edge cases, create one item: "Are edge cases X, Y, Z addressed in requirements? [Coverage]"
|
|
||||||
|
|
||||||
**🚫 ABSOLUTELY PROHIBITED** - These make it an implementation test, not a requirements test:
|
|
||||||
- ❌ Any item starting with "Verify", "Test", "Confirm", "Check" + implementation behavior
|
|
||||||
- ❌ References to code execution, user actions, system behavior
|
|
||||||
- ❌ "Displays correctly", "works properly", "functions as expected"
|
|
||||||
- ❌ "Click", "navigate", "render", "load", "execute"
|
|
||||||
- ❌ Test cases, test plans, QA procedures
|
|
||||||
- ❌ Implementation details (frameworks, APIs, algorithms)
|
|
||||||
|
|
||||||
**✅ REQUIRED PATTERNS** - These test requirements quality:
|
|
||||||
- ✅ "Are [requirement type] defined/specified/documented for [scenario]?"
|
|
||||||
- ✅ "Is [vague term] quantified/clarified with specific criteria?"
|
|
||||||
- ✅ "Are requirements consistent between [section A] and [section B]?"
|
|
||||||
- ✅ "Can [requirement] be objectively measured/verified?"
|
|
||||||
- ✅ "Are [edge cases/scenarios] addressed in requirements?"
|
|
||||||
- ✅ "Does the spec define [missing aspect]?"
|
|
||||||
|
|
||||||
6. **Structure Reference**: Generate the checklist following the canonical template in `.specify/templates/checklist-template.md` for title, meta section, category headings, and ID formatting. If template is unavailable, use: H1 title, purpose/created meta lines, `##` category sections containing `- [ ] CHK### <requirement item>` lines with globally incrementing IDs starting at CHK001.
|
|
||||||
|
|
||||||
7. **Report**: Output full path to created checklist, item count, and remind user that each run creates a new file. Summarize:
|
|
||||||
- Focus areas selected
|
|
||||||
- Depth level
|
|
||||||
- Actor/timing
|
|
||||||
- Any explicit user-specified must-have items incorporated
|
|
||||||
|
|
||||||
**Important**: Each `/speckit.checklist` command invocation creates a checklist file using short, descriptive names unless file already exists. This allows:
|
|
||||||
|
|
||||||
- Multiple checklists of different types (e.g., `ux.md`, `test.md`, `security.md`)
|
|
||||||
- Simple, memorable filenames that indicate checklist purpose
|
|
||||||
- Easy identification and navigation in the `checklists/` folder
|
|
||||||
|
|
||||||
To avoid clutter, use descriptive types and clean up obsolete checklists when done.
|
|
||||||
|
|
||||||
## Example Checklist Types & Sample Items
|
|
||||||
|
|
||||||
**UX Requirements Quality:** `ux.md`
|
|
||||||
|
|
||||||
Sample items (testing the requirements, NOT the implementation):
|
|
||||||
|
|
||||||
- "Are visual hierarchy requirements defined with measurable criteria? [Clarity, Spec §FR-1]"
|
|
||||||
- "Is the number and positioning of UI elements explicitly specified? [Completeness, Spec §FR-1]"
|
|
||||||
- "Are interaction state requirements (hover, focus, active) consistently defined? [Consistency]"
|
|
||||||
- "Are accessibility requirements specified for all interactive elements? [Coverage, Gap]"
|
|
||||||
- "Is fallback behavior defined when images fail to load? [Edge Case, Gap]"
|
|
||||||
- "Can 'prominent display' be objectively measured? [Measurability, Spec §FR-4]"
|
|
||||||
|
|
||||||
**API Requirements Quality:** `api.md`
|
|
||||||
|
|
||||||
Sample items:
|
|
||||||
|
|
||||||
- "Are error response formats specified for all failure scenarios? [Completeness]"
|
|
||||||
- "Are rate limiting requirements quantified with specific thresholds? [Clarity]"
|
|
||||||
- "Are authentication requirements consistent across all endpoints? [Consistency]"
|
|
||||||
- "Are retry/timeout requirements defined for external dependencies? [Coverage, Gap]"
|
|
||||||
- "Is versioning strategy documented in requirements? [Gap]"
|
|
||||||
|
|
||||||
**Performance Requirements Quality:** `performance.md`
|
|
||||||
|
|
||||||
Sample items:
|
|
||||||
|
|
||||||
- "Are performance requirements quantified with specific metrics? [Clarity]"
|
|
||||||
- "Are performance targets defined for all critical user journeys? [Coverage]"
|
|
||||||
- "Are performance requirements under different load conditions specified? [Completeness]"
|
|
||||||
- "Can performance requirements be objectively measured? [Measurability]"
|
|
||||||
- "Are degradation requirements defined for high-load scenarios? [Edge Case, Gap]"
|
|
||||||
|
|
||||||
**Security Requirements Quality:** `security.md`
|
|
||||||
|
|
||||||
Sample items:
|
|
||||||
|
|
||||||
- "Are authentication requirements specified for all protected resources? [Coverage]"
|
|
||||||
- "Are data protection requirements defined for sensitive information? [Completeness]"
|
|
||||||
- "Is the threat model documented and requirements aligned to it? [Traceability]"
|
|
||||||
- "Are security requirements consistent with compliance obligations? [Consistency]"
|
|
||||||
- "Are security failure/breach response requirements defined? [Gap, Exception Flow]"
|
|
||||||
|
|
||||||
## Anti-Examples: What NOT To Do
|
|
||||||
|
|
||||||
**❌ WRONG - These test implementation, not requirements:**
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
- [ ] CHK001 - Verify landing page displays 3 episode cards [Spec §FR-001]
|
|
||||||
- [ ] CHK002 - Test hover states work correctly on desktop [Spec §FR-003]
|
|
||||||
- [ ] CHK003 - Confirm logo click navigates to home page [Spec §FR-010]
|
|
||||||
- [ ] CHK004 - Check that related episodes section shows 3-5 items [Spec §FR-005]
|
|
||||||
```
|
|
||||||
|
|
||||||
**✅ CORRECT - These test requirements quality:**
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
- [ ] CHK001 - Are the number and layout of featured episodes explicitly specified? [Completeness, Spec §FR-001]
|
|
||||||
- [ ] CHK002 - Are hover state requirements consistently defined for all interactive elements? [Consistency, Spec §FR-003]
|
|
||||||
- [ ] CHK003 - Are navigation requirements clear for all clickable brand elements? [Clarity, Spec §FR-010]
|
|
||||||
- [ ] CHK004 - Is the selection criteria for related episodes documented? [Gap, Spec §FR-005]
|
|
||||||
- [ ] CHK005 - Are loading state requirements defined for asynchronous episode data? [Gap]
|
|
||||||
- [ ] CHK006 - Can "visual hierarchy" requirements be objectively measured? [Measurability, Spec §FR-001]
|
|
||||||
```
|
|
||||||
|
|
||||||
**Key Differences:**
|
|
||||||
|
|
||||||
- Wrong: Tests if the system works correctly
|
|
||||||
- Correct: Tests if the requirements are written correctly
|
|
||||||
- Wrong: Verification of behavior
|
|
||||||
- Correct: Validation of requirement quality
|
|
||||||
- Wrong: "Does it do X?"
|
|
||||||
- Correct: "Is X clearly specified?"
|
|
||||||
@@ -1,181 +0,0 @@
|
|||||||
---
|
|
||||||
description: Identify underspecified areas in the current feature spec by asking up to 5 highly targeted clarification questions and encoding answers back into the spec.
|
|
||||||
handoffs:
|
|
||||||
- label: Build Technical Plan
|
|
||||||
agent: speckit.plan
|
|
||||||
prompt: Create a plan for the spec. I am building with...
|
|
||||||
---
|
|
||||||
|
|
||||||
## User Input
|
|
||||||
|
|
||||||
```text
|
|
||||||
$ARGUMENTS
|
|
||||||
```
|
|
||||||
|
|
||||||
You **MUST** consider the user input before proceeding (if not empty).
|
|
||||||
|
|
||||||
## Outline
|
|
||||||
|
|
||||||
Goal: Detect and reduce ambiguity or missing decision points in the active feature specification and record the clarifications directly in the spec file.
|
|
||||||
|
|
||||||
Note: This clarification workflow is expected to run (and be completed) BEFORE invoking `/speckit.plan`. If the user explicitly states they are skipping clarification (e.g., exploratory spike), you may proceed, but must warn that downstream rework risk increases.
|
|
||||||
|
|
||||||
Execution steps:
|
|
||||||
|
|
||||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --paths-only` from repo root **once** (combined `--json --paths-only` mode / `-Json -PathsOnly`). Parse minimal JSON payload fields:
|
|
||||||
- `FEATURE_DIR`
|
|
||||||
- `FEATURE_SPEC`
|
|
||||||
- (Optionally capture `IMPL_PLAN`, `TASKS` for future chained flows.)
|
|
||||||
- If JSON parsing fails, abort and instruct user to re-run `/speckit.specify` or verify feature branch environment.
|
|
||||||
- For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
|
||||||
|
|
||||||
2. Load the current spec file. Perform a structured ambiguity & coverage scan using this taxonomy. For each category, mark status: Clear / Partial / Missing. Produce an internal coverage map used for prioritization (do not output raw map unless no questions will be asked).
|
|
||||||
|
|
||||||
Functional Scope & Behavior:
|
|
||||||
- Core user goals & success criteria
|
|
||||||
- Explicit out-of-scope declarations
|
|
||||||
- User roles / personas differentiation
|
|
||||||
|
|
||||||
Domain & Data Model:
|
|
||||||
- Entities, attributes, relationships
|
|
||||||
- Identity & uniqueness rules
|
|
||||||
- Lifecycle/state transitions
|
|
||||||
- Data volume / scale assumptions
|
|
||||||
|
|
||||||
Interaction & UX Flow:
|
|
||||||
- Critical user journeys / sequences
|
|
||||||
- Error/empty/loading states
|
|
||||||
- Accessibility or localization notes
|
|
||||||
|
|
||||||
Non-Functional Quality Attributes:
|
|
||||||
- Performance (latency, throughput targets)
|
|
||||||
- Scalability (horizontal/vertical, limits)
|
|
||||||
- Reliability & availability (uptime, recovery expectations)
|
|
||||||
- Observability (logging, metrics, tracing signals)
|
|
||||||
- Security & privacy (authN/Z, data protection, threat assumptions)
|
|
||||||
- Compliance / regulatory constraints (if any)
|
|
||||||
|
|
||||||
Integration & External Dependencies:
|
|
||||||
- External services/APIs and failure modes
|
|
||||||
- Data import/export formats
|
|
||||||
- Protocol/versioning assumptions
|
|
||||||
|
|
||||||
Edge Cases & Failure Handling:
|
|
||||||
- Negative scenarios
|
|
||||||
- Rate limiting / throttling
|
|
||||||
- Conflict resolution (e.g., concurrent edits)
|
|
||||||
|
|
||||||
Constraints & Tradeoffs:
|
|
||||||
- Technical constraints (language, storage, hosting)
|
|
||||||
- Explicit tradeoffs or rejected alternatives
|
|
||||||
|
|
||||||
Terminology & Consistency:
|
|
||||||
- Canonical glossary terms
|
|
||||||
- Avoided synonyms / deprecated terms
|
|
||||||
|
|
||||||
Completion Signals:
|
|
||||||
- Acceptance criteria testability
|
|
||||||
- Measurable Definition of Done style indicators
|
|
||||||
|
|
||||||
Misc / Placeholders:
|
|
||||||
- TODO markers / unresolved decisions
|
|
||||||
- Ambiguous adjectives ("robust", "intuitive") lacking quantification
|
|
||||||
|
|
||||||
For each category with Partial or Missing status, add a candidate question opportunity unless:
|
|
||||||
- Clarification would not materially change implementation or validation strategy
|
|
||||||
- Information is better deferred to planning phase (note internally)
|
|
||||||
|
|
||||||
3. Generate (internally) a prioritized queue of candidate clarification questions (maximum 5). Do NOT output them all at once. Apply these constraints:
|
|
||||||
- Maximum of 10 total questions across the whole session.
|
|
||||||
- Each question must be answerable with EITHER:
|
|
||||||
- A short multiple‑choice selection (2–5 distinct, mutually exclusive options), OR
|
|
||||||
- A one-word / short‑phrase answer (explicitly constrain: "Answer in <=5 words").
|
|
||||||
- Only include questions whose answers materially impact architecture, data modeling, task decomposition, test design, UX behavior, operational readiness, or compliance validation.
|
|
||||||
- Ensure category coverage balance: attempt to cover the highest impact unresolved categories first; avoid asking two low-impact questions when a single high-impact area (e.g., security posture) is unresolved.
|
|
||||||
- Exclude questions already answered, trivial stylistic preferences, or plan-level execution details (unless blocking correctness).
|
|
||||||
- Favor clarifications that reduce downstream rework risk or prevent misaligned acceptance tests.
|
|
||||||
- If more than 5 categories remain unresolved, select the top 5 by (Impact * Uncertainty) heuristic.
|
|
||||||
|
|
||||||
4. Sequential questioning loop (interactive):
|
|
||||||
- Present EXACTLY ONE question at a time.
|
|
||||||
- For multiple‑choice questions:
|
|
||||||
- **Analyze all options** and determine the **most suitable option** based on:
|
|
||||||
- Best practices for the project type
|
|
||||||
- Common patterns in similar implementations
|
|
||||||
- Risk reduction (security, performance, maintainability)
|
|
||||||
- Alignment with any explicit project goals or constraints visible in the spec
|
|
||||||
- Present your **recommended option prominently** at the top with clear reasoning (1-2 sentences explaining why this is the best choice).
|
|
||||||
- Format as: `**Recommended:** Option [X] - <reasoning>`
|
|
||||||
- Then render all options as a Markdown table:
|
|
||||||
|
|
||||||
| Option | Description |
|
|
||||||
|--------|-------------|
|
|
||||||
| A | <Option A description> |
|
|
||||||
| B | <Option B description> |
|
|
||||||
| C | <Option C description> (add D/E as needed up to 5) |
|
|
||||||
| Short | Provide a different short answer (<=5 words) (Include only if free-form alternative is appropriate) |
|
|
||||||
|
|
||||||
- After the table, add: `You can reply with the option letter (e.g., "A"), accept the recommendation by saying "yes" or "recommended", or provide your own short answer.`
|
|
||||||
- For short‑answer style (no meaningful discrete options):
|
|
||||||
- Provide your **suggested answer** based on best practices and context.
|
|
||||||
- Format as: `**Suggested:** <your proposed answer> - <brief reasoning>`
|
|
||||||
- Then output: `Format: Short answer (<=5 words). You can accept the suggestion by saying "yes" or "suggested", or provide your own answer.`
|
|
||||||
- After the user answers:
|
|
||||||
- If the user replies with "yes", "recommended", or "suggested", use your previously stated recommendation/suggestion as the answer.
|
|
||||||
- Otherwise, validate the answer maps to one option or fits the <=5 word constraint.
|
|
||||||
- If ambiguous, ask for a quick disambiguation (count still belongs to same question; do not advance).
|
|
||||||
- Once satisfactory, record it in working memory (do not yet write to disk) and move to the next queued question.
|
|
||||||
- Stop asking further questions when:
|
|
||||||
- All critical ambiguities resolved early (remaining queued items become unnecessary), OR
|
|
||||||
- User signals completion ("done", "good", "no more"), OR
|
|
||||||
- You reach 5 asked questions.
|
|
||||||
- Never reveal future queued questions in advance.
|
|
||||||
- If no valid questions exist at start, immediately report no critical ambiguities.
|
|
||||||
|
|
||||||
5. Integration after EACH accepted answer (incremental update approach):
|
|
||||||
- Maintain in-memory representation of the spec (loaded once at start) plus the raw file contents.
|
|
||||||
- For the first integrated answer in this session:
|
|
||||||
- Ensure a `## Clarifications` section exists (create it just after the highest-level contextual/overview section per the spec template if missing).
|
|
||||||
- Under it, create (if not present) a `### Session YYYY-MM-DD` subheading for today.
|
|
||||||
- Append a bullet line immediately after acceptance: `- Q: <question> → A: <final answer>`.
|
|
||||||
- Then immediately apply the clarification to the most appropriate section(s):
|
|
||||||
- Functional ambiguity → Update or add a bullet in Functional Requirements.
|
|
||||||
- User interaction / actor distinction → Update User Stories or Actors subsection (if present) with clarified role, constraint, or scenario.
|
|
||||||
- Data shape / entities → Update Data Model (add fields, types, relationships) preserving ordering; note added constraints succinctly.
|
|
||||||
- Non-functional constraint → Add/modify measurable criteria in Non-Functional / Quality Attributes section (convert vague adjective to metric or explicit target).
|
|
||||||
- Edge case / negative flow → Add a new bullet under Edge Cases / Error Handling (or create such subsection if template provides placeholder for it).
|
|
||||||
- Terminology conflict → Normalize term across spec; retain original only if necessary by adding `(formerly referred to as "X")` once.
|
|
||||||
- If the clarification invalidates an earlier ambiguous statement, replace that statement instead of duplicating; leave no obsolete contradictory text.
|
|
||||||
- Save the spec file AFTER each integration to minimize risk of context loss (atomic overwrite).
|
|
||||||
- Preserve formatting: do not reorder unrelated sections; keep heading hierarchy intact.
|
|
||||||
- Keep each inserted clarification minimal and testable (avoid narrative drift).
|
|
||||||
|
|
||||||
6. Validation (performed after EACH write plus final pass):
|
|
||||||
- Clarifications session contains exactly one bullet per accepted answer (no duplicates).
|
|
||||||
- Total asked (accepted) questions ≤ 5.
|
|
||||||
- Updated sections contain no lingering vague placeholders the new answer was meant to resolve.
|
|
||||||
- No contradictory earlier statement remains (scan for now-invalid alternative choices removed).
|
|
||||||
- Markdown structure valid; only allowed new headings: `## Clarifications`, `### Session YYYY-MM-DD`.
|
|
||||||
- Terminology consistency: same canonical term used across all updated sections.
|
|
||||||
|
|
||||||
7. Write the updated spec back to `FEATURE_SPEC`.
|
|
||||||
|
|
||||||
8. Report completion (after questioning loop ends or early termination):
|
|
||||||
- Number of questions asked & answered.
|
|
||||||
- Path to updated spec.
|
|
||||||
- Sections touched (list names).
|
|
||||||
- Coverage summary table listing each taxonomy category with Status: Resolved (was Partial/Missing and addressed), Deferred (exceeds question quota or better suited for planning), Clear (already sufficient), Outstanding (still Partial/Missing but low impact).
|
|
||||||
- If any Outstanding or Deferred remain, recommend whether to proceed to `/speckit.plan` or run `/speckit.clarify` again later post-plan.
|
|
||||||
- Suggested next command.
|
|
||||||
|
|
||||||
Behavior rules:
|
|
||||||
|
|
||||||
- If no meaningful ambiguities found (or all potential questions would be low-impact), respond: "No critical ambiguities detected worth formal clarification." and suggest proceeding.
|
|
||||||
- If spec file missing, instruct user to run `/speckit.specify` first (do not create a new spec here).
|
|
||||||
- Never exceed 5 total asked questions (clarification retries for a single question do not count as new questions).
|
|
||||||
- Avoid speculative tech stack questions unless the absence blocks functional clarity.
|
|
||||||
- Respect user early termination signals ("stop", "done", "proceed").
|
|
||||||
- If no questions asked due to full coverage, output a compact coverage summary (all categories Clear) then suggest advancing.
|
|
||||||
- If quota reached with unresolved high-impact categories remaining, explicitly flag them under Deferred with rationale.
|
|
||||||
|
|
||||||
Context for prioritization: $ARGUMENTS
|
|
||||||
@@ -1,82 +0,0 @@
|
|||||||
---
|
|
||||||
description: Create or update the project constitution from interactive or provided principle inputs, ensuring all dependent templates stay in sync.
|
|
||||||
handoffs:
|
|
||||||
- label: Build Specification
|
|
||||||
agent: speckit.specify
|
|
||||||
prompt: Implement the feature specification based on the updated constitution. I want to build...
|
|
||||||
---
|
|
||||||
|
|
||||||
## User Input
|
|
||||||
|
|
||||||
```text
|
|
||||||
$ARGUMENTS
|
|
||||||
```
|
|
||||||
|
|
||||||
You **MUST** consider the user input before proceeding (if not empty).
|
|
||||||
|
|
||||||
## Outline
|
|
||||||
|
|
||||||
You are updating the project constitution at `.specify/memory/constitution.md`. This file is a TEMPLATE containing placeholder tokens in square brackets (e.g. `[PROJECT_NAME]`, `[PRINCIPLE_1_NAME]`). Your job is to (a) collect/derive concrete values, (b) fill the template precisely, and (c) propagate any amendments across dependent artifacts.
|
|
||||||
|
|
||||||
Follow this execution flow:
|
|
||||||
|
|
||||||
1. Load the existing constitution template at `.specify/memory/constitution.md`.
|
|
||||||
- Identify every placeholder token of the form `[ALL_CAPS_IDENTIFIER]`.
|
|
||||||
**IMPORTANT**: The user might require less or more principles than the ones used in the template. If a number is specified, respect that - follow the general template. You will update the doc accordingly.
|
|
||||||
|
|
||||||
2. Collect/derive values for placeholders:
|
|
||||||
- If user input (conversation) supplies a value, use it.
|
|
||||||
- Otherwise infer from existing repo context (README, docs, prior constitution versions if embedded).
|
|
||||||
- For governance dates: `RATIFICATION_DATE` is the original adoption date (if unknown ask or mark TODO), `LAST_AMENDED_DATE` is today if changes are made, otherwise keep previous.
|
|
||||||
- `CONSTITUTION_VERSION` must increment according to semantic versioning rules:
|
|
||||||
- MAJOR: Backward incompatible governance/principle removals or redefinitions.
|
|
||||||
- MINOR: New principle/section added or materially expanded guidance.
|
|
||||||
- PATCH: Clarifications, wording, typo fixes, non-semantic refinements.
|
|
||||||
- If version bump type ambiguous, propose reasoning before finalizing.
|
|
||||||
|
|
||||||
3. Draft the updated constitution content:
|
|
||||||
- Replace every placeholder with concrete text (no bracketed tokens left except intentionally retained template slots that the project has chosen not to define yet—explicitly justify any left).
|
|
||||||
- Preserve heading hierarchy and comments can be removed once replaced unless they still add clarifying guidance.
|
|
||||||
- Ensure each Principle section: succinct name line, paragraph (or bullet list) capturing non‑negotiable rules, explicit rationale if not obvious.
|
|
||||||
- Ensure Governance section lists amendment procedure, versioning policy, and compliance review expectations.
|
|
||||||
|
|
||||||
4. Consistency propagation checklist (convert prior checklist into active validations):
|
|
||||||
- Read `.specify/templates/plan-template.md` and ensure any "Constitution Check" or rules align with updated principles.
|
|
||||||
- Read `.specify/templates/spec-template.md` for scope/requirements alignment—update if constitution adds/removes mandatory sections or constraints.
|
|
||||||
- Read `.specify/templates/tasks-template.md` and ensure task categorization reflects new or removed principle-driven task types (e.g., observability, versioning, testing discipline).
|
|
||||||
- Read each command file in `.specify/templates/commands/*.md` (including this one) to verify no outdated references (agent-specific names like CLAUDE only) remain when generic guidance is required.
|
|
||||||
- Read any runtime guidance docs (e.g., `README.md`, `docs/quickstart.md`, or agent-specific guidance files if present). Update references to principles changed.
|
|
||||||
|
|
||||||
5. Produce a Sync Impact Report (prepend as an HTML comment at top of the constitution file after update):
|
|
||||||
- Version change: old → new
|
|
||||||
- List of modified principles (old title → new title if renamed)
|
|
||||||
- Added sections
|
|
||||||
- Removed sections
|
|
||||||
- Templates requiring updates (✅ updated / ⚠ pending) with file paths
|
|
||||||
- Follow-up TODOs if any placeholders intentionally deferred.
|
|
||||||
|
|
||||||
6. Validation before final output:
|
|
||||||
- No remaining unexplained bracket tokens.
|
|
||||||
- Version line matches report.
|
|
||||||
- Dates ISO format YYYY-MM-DD.
|
|
||||||
- Principles are declarative, testable, and free of vague language ("should" → replace with MUST/SHOULD rationale where appropriate).
|
|
||||||
|
|
||||||
7. Write the completed constitution back to `.specify/memory/constitution.md` (overwrite).
|
|
||||||
|
|
||||||
8. Output a final summary to the user with:
|
|
||||||
- New version and bump rationale.
|
|
||||||
- Any files flagged for manual follow-up.
|
|
||||||
- Suggested commit message (e.g., `docs: amend constitution to vX.Y.Z (principle additions + governance update)`).
|
|
||||||
|
|
||||||
Formatting & Style Requirements:
|
|
||||||
|
|
||||||
- Use Markdown headings exactly as in the template (do not demote/promote levels).
|
|
||||||
- Wrap long rationale lines to keep readability (<100 chars ideally) but do not hard enforce with awkward breaks.
|
|
||||||
- Keep a single blank line between sections.
|
|
||||||
- Avoid trailing whitespace.
|
|
||||||
|
|
||||||
If the user supplies partial updates (e.g., only one principle revision), still perform validation and version decision steps.
|
|
||||||
|
|
||||||
If critical info missing (e.g., ratification date truly unknown), insert `TODO(<FIELD_NAME>): explanation` and include in the Sync Impact Report under deferred items.
|
|
||||||
|
|
||||||
Do not create a new template; always operate on the existing `.specify/memory/constitution.md` file.
|
|
||||||
@@ -1,135 +0,0 @@
|
|||||||
---
|
|
||||||
description: Execute the implementation plan by processing and executing all tasks defined in tasks.md
|
|
||||||
---
|
|
||||||
|
|
||||||
## User Input
|
|
||||||
|
|
||||||
```text
|
|
||||||
$ARGUMENTS
|
|
||||||
```
|
|
||||||
|
|
||||||
You **MUST** consider the user input before proceeding (if not empty).
|
|
||||||
|
|
||||||
## Outline
|
|
||||||
|
|
||||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
|
||||||
|
|
||||||
2. **Check checklists status** (if FEATURE_DIR/checklists/ exists):
|
|
||||||
- Scan all checklist files in the checklists/ directory
|
|
||||||
- For each checklist, count:
|
|
||||||
- Total items: All lines matching `- [ ]` or `- [X]` or `- [x]`
|
|
||||||
- Completed items: Lines matching `- [X]` or `- [x]`
|
|
||||||
- Incomplete items: Lines matching `- [ ]`
|
|
||||||
- Create a status table:
|
|
||||||
|
|
||||||
```text
|
|
||||||
| Checklist | Total | Completed | Incomplete | Status |
|
|
||||||
|-----------|-------|-----------|------------|--------|
|
|
||||||
| ux.md | 12 | 12 | 0 | ✓ PASS |
|
|
||||||
| test.md | 8 | 5 | 3 | ✗ FAIL |
|
|
||||||
| security.md | 6 | 6 | 0 | ✓ PASS |
|
|
||||||
```
|
|
||||||
|
|
||||||
- Calculate overall status:
|
|
||||||
- **PASS**: All checklists have 0 incomplete items
|
|
||||||
- **FAIL**: One or more checklists have incomplete items
|
|
||||||
|
|
||||||
- **If any checklist is incomplete**:
|
|
||||||
- Display the table with incomplete item counts
|
|
||||||
- **STOP** and ask: "Some checklists are incomplete. Do you want to proceed with implementation anyway? (yes/no)"
|
|
||||||
- Wait for user response before continuing
|
|
||||||
- If user says "no" or "wait" or "stop", halt execution
|
|
||||||
- If user says "yes" or "proceed" or "continue", proceed to step 3
|
|
||||||
|
|
||||||
- **If all checklists are complete**:
|
|
||||||
- Display the table showing all checklists passed
|
|
||||||
- Automatically proceed to step 3
|
|
||||||
|
|
||||||
3. Load and analyze the implementation context:
|
|
||||||
- **REQUIRED**: Read tasks.md for the complete task list and execution plan
|
|
||||||
- **REQUIRED**: Read plan.md for tech stack, architecture, and file structure
|
|
||||||
- **IF EXISTS**: Read data-model.md for entities and relationships
|
|
||||||
- **IF EXISTS**: Read contracts/ for API specifications and test requirements
|
|
||||||
- **IF EXISTS**: Read research.md for technical decisions and constraints
|
|
||||||
- **IF EXISTS**: Read quickstart.md for integration scenarios
|
|
||||||
|
|
||||||
4. **Project Setup Verification**:
|
|
||||||
- **REQUIRED**: Create/verify ignore files based on actual project setup:
|
|
||||||
|
|
||||||
**Detection & Creation Logic**:
|
|
||||||
- Check if the following command succeeds to determine if the repository is a git repo (create/verify .gitignore if so):
|
|
||||||
|
|
||||||
```sh
|
|
||||||
git rev-parse --git-dir 2>/dev/null
|
|
||||||
```
|
|
||||||
|
|
||||||
- Check if Dockerfile* exists or Docker in plan.md → create/verify .dockerignore
|
|
||||||
- Check if .eslintrc* exists → create/verify .eslintignore
|
|
||||||
- Check if eslint.config.* exists → ensure the config's `ignores` entries cover required patterns
|
|
||||||
- Check if .prettierrc* exists → create/verify .prettierignore
|
|
||||||
- Check if .npmrc or package.json exists → create/verify .npmignore (if publishing)
|
|
||||||
- Check if terraform files (*.tf) exist → create/verify .terraformignore
|
|
||||||
- Check if .helmignore needed (helm charts present) → create/verify .helmignore
|
|
||||||
|
|
||||||
**If ignore file already exists**: Verify it contains essential patterns, append missing critical patterns only
|
|
||||||
**If ignore file missing**: Create with full pattern set for detected technology
|
|
||||||
|
|
||||||
**Common Patterns by Technology** (from plan.md tech stack):
|
|
||||||
- **Node.js/JavaScript/TypeScript**: `node_modules/`, `dist/`, `build/`, `*.log`, `.env*`
|
|
||||||
- **Python**: `__pycache__/`, `*.pyc`, `.venv/`, `venv/`, `dist/`, `*.egg-info/`
|
|
||||||
- **Java**: `target/`, `*.class`, `*.jar`, `.gradle/`, `build/`
|
|
||||||
- **C#/.NET**: `bin/`, `obj/`, `*.user`, `*.suo`, `packages/`
|
|
||||||
- **Go**: `*.exe`, `*.test`, `vendor/`, `*.out`
|
|
||||||
- **Ruby**: `.bundle/`, `log/`, `tmp/`, `*.gem`, `vendor/bundle/`
|
|
||||||
- **PHP**: `vendor/`, `*.log`, `*.cache`, `*.env`
|
|
||||||
- **Rust**: `target/`, `debug/`, `release/`, `*.rs.bk`, `*.rlib`, `*.prof*`, `.idea/`, `*.log`, `.env*`
|
|
||||||
- **Kotlin**: `build/`, `out/`, `.gradle/`, `.idea/`, `*.class`, `*.jar`, `*.iml`, `*.log`, `.env*`
|
|
||||||
- **C++**: `build/`, `bin/`, `obj/`, `out/`, `*.o`, `*.so`, `*.a`, `*.exe`, `*.dll`, `.idea/`, `*.log`, `.env*`
|
|
||||||
- **C**: `build/`, `bin/`, `obj/`, `out/`, `*.o`, `*.a`, `*.so`, `*.exe`, `Makefile`, `config.log`, `.idea/`, `*.log`, `.env*`
|
|
||||||
- **Swift**: `.build/`, `DerivedData/`, `*.swiftpm/`, `Packages/`
|
|
||||||
- **R**: `.Rproj.user/`, `.Rhistory`, `.RData`, `.Ruserdata`, `*.Rproj`, `packrat/`, `renv/`
|
|
||||||
- **Universal**: `.DS_Store`, `Thumbs.db`, `*.tmp`, `*.swp`, `.vscode/`, `.idea/`
|
|
||||||
|
|
||||||
**Tool-Specific Patterns**:
|
|
||||||
- **Docker**: `node_modules/`, `.git/`, `Dockerfile*`, `.dockerignore`, `*.log*`, `.env*`, `coverage/`
|
|
||||||
- **ESLint**: `node_modules/`, `dist/`, `build/`, `coverage/`, `*.min.js`
|
|
||||||
- **Prettier**: `node_modules/`, `dist/`, `build/`, `coverage/`, `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`
|
|
||||||
- **Terraform**: `.terraform/`, `*.tfstate*`, `*.tfvars`, `.terraform.lock.hcl`
|
|
||||||
- **Kubernetes/k8s**: `*.secret.yaml`, `secrets/`, `.kube/`, `kubeconfig*`, `*.key`, `*.crt`
|
|
||||||
|
|
||||||
5. Parse tasks.md structure and extract:
|
|
||||||
- **Task phases**: Setup, Tests, Core, Integration, Polish
|
|
||||||
- **Task dependencies**: Sequential vs parallel execution rules
|
|
||||||
- **Task details**: ID, description, file paths, parallel markers [P]
|
|
||||||
- **Execution flow**: Order and dependency requirements
|
|
||||||
|
|
||||||
6. Execute implementation following the task plan:
|
|
||||||
- **Phase-by-phase execution**: Complete each phase before moving to the next
|
|
||||||
- **Respect dependencies**: Run sequential tasks in order, parallel tasks [P] can run together
|
|
||||||
- **Follow TDD approach**: Execute test tasks before their corresponding implementation tasks
|
|
||||||
- **File-based coordination**: Tasks affecting the same files must run sequentially
|
|
||||||
- **Validation checkpoints**: Verify each phase completion before proceeding
|
|
||||||
|
|
||||||
7. Implementation execution rules:
|
|
||||||
- **Setup first**: Initialize project structure, dependencies, configuration
|
|
||||||
- **Tests before code**: If you need to write tests for contracts, entities, and integration scenarios
|
|
||||||
- **Core development**: Implement models, services, CLI commands, endpoints
|
|
||||||
- **Integration work**: Database connections, middleware, logging, external services
|
|
||||||
- **Polish and validation**: Unit tests, performance optimization, documentation
|
|
||||||
|
|
||||||
8. Progress tracking and error handling:
|
|
||||||
- Report progress after each completed task
|
|
||||||
- Halt execution if any non-parallel task fails
|
|
||||||
- For parallel tasks [P], continue with successful tasks, report failed ones
|
|
||||||
- Provide clear error messages with context for debugging
|
|
||||||
- Suggest next steps if implementation cannot proceed
|
|
||||||
- **IMPORTANT** For completed tasks, make sure to mark the task off as [X] in the tasks file.
|
|
||||||
|
|
||||||
9. Completion validation:
|
|
||||||
- Verify all required tasks are completed
|
|
||||||
- Check that implemented features match the original specification
|
|
||||||
- Validate that tests pass and coverage meets requirements
|
|
||||||
- Confirm the implementation follows the technical plan
|
|
||||||
- Report final status with summary of completed work
|
|
||||||
|
|
||||||
Note: This command assumes a complete task breakdown exists in tasks.md. If tasks are incomplete or missing, suggest running `/speckit.tasks` first to regenerate the task list.
|
|
||||||
@@ -1,89 +0,0 @@
|
|||||||
---
|
|
||||||
description: Execute the implementation planning workflow using the plan template to generate design artifacts.
|
|
||||||
handoffs:
|
|
||||||
- label: Create Tasks
|
|
||||||
agent: speckit.tasks
|
|
||||||
prompt: Break the plan into tasks
|
|
||||||
send: true
|
|
||||||
- label: Create Checklist
|
|
||||||
agent: speckit.checklist
|
|
||||||
prompt: Create a checklist for the following domain...
|
|
||||||
---
|
|
||||||
|
|
||||||
## User Input
|
|
||||||
|
|
||||||
```text
|
|
||||||
$ARGUMENTS
|
|
||||||
```
|
|
||||||
|
|
||||||
You **MUST** consider the user input before proceeding (if not empty).
|
|
||||||
|
|
||||||
## Outline
|
|
||||||
|
|
||||||
1. **Setup**: Run `.specify/scripts/bash/setup-plan.sh --json` from repo root and parse JSON for FEATURE_SPEC, IMPL_PLAN, SPECS_DIR, BRANCH. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
|
||||||
|
|
||||||
2. **Load context**: Read FEATURE_SPEC and `.specify/memory/constitution.md`. Load IMPL_PLAN template (already copied).
|
|
||||||
|
|
||||||
3. **Execute plan workflow**: Follow the structure in IMPL_PLAN template to:
|
|
||||||
- Fill Technical Context (mark unknowns as "NEEDS CLARIFICATION")
|
|
||||||
- Fill Constitution Check section from constitution
|
|
||||||
- Evaluate gates (ERROR if violations unjustified)
|
|
||||||
- Phase 0: Generate research.md (resolve all NEEDS CLARIFICATION)
|
|
||||||
- Phase 1: Generate data-model.md, contracts/, quickstart.md
|
|
||||||
- Phase 1: Update agent context by running the agent script
|
|
||||||
- Re-evaluate Constitution Check post-design
|
|
||||||
|
|
||||||
4. **Stop and report**: Command ends after Phase 2 planning. Report branch, IMPL_PLAN path, and generated artifacts.
|
|
||||||
|
|
||||||
## Phases
|
|
||||||
|
|
||||||
### Phase 0: Outline & Research
|
|
||||||
|
|
||||||
1. **Extract unknowns from Technical Context** above:
|
|
||||||
- For each NEEDS CLARIFICATION → research task
|
|
||||||
- For each dependency → best practices task
|
|
||||||
- For each integration → patterns task
|
|
||||||
|
|
||||||
2. **Generate and dispatch research agents**:
|
|
||||||
|
|
||||||
```text
|
|
||||||
For each unknown in Technical Context:
|
|
||||||
Task: "Research {unknown} for {feature context}"
|
|
||||||
For each technology choice:
|
|
||||||
Task: "Find best practices for {tech} in {domain}"
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Consolidate findings** in `research.md` using format:
|
|
||||||
- Decision: [what was chosen]
|
|
||||||
- Rationale: [why chosen]
|
|
||||||
- Alternatives considered: [what else evaluated]
|
|
||||||
|
|
||||||
**Output**: research.md with all NEEDS CLARIFICATION resolved
|
|
||||||
|
|
||||||
### Phase 1: Design & Contracts
|
|
||||||
|
|
||||||
**Prerequisites:** `research.md` complete
|
|
||||||
|
|
||||||
1. **Extract entities from feature spec** → `data-model.md`:
|
|
||||||
- Entity name, fields, relationships
|
|
||||||
- Validation rules from requirements
|
|
||||||
- State transitions if applicable
|
|
||||||
|
|
||||||
2. **Generate API contracts** from functional requirements:
|
|
||||||
- For each user action → endpoint
|
|
||||||
- Use standard REST/GraphQL patterns
|
|
||||||
- Output OpenAPI/GraphQL schema to `/contracts/`
|
|
||||||
|
|
||||||
3. **Agent context update**:
|
|
||||||
- Run `.specify/scripts/bash/update-agent-context.sh claude`
|
|
||||||
- These scripts detect which AI agent is in use
|
|
||||||
- Update the appropriate agent-specific context file
|
|
||||||
- Add only new technology from current plan
|
|
||||||
- Preserve manual additions between markers
|
|
||||||
|
|
||||||
**Output**: data-model.md, /contracts/*, quickstart.md, agent-specific file
|
|
||||||
|
|
||||||
## Key rules
|
|
||||||
|
|
||||||
- Use absolute paths
|
|
||||||
- ERROR on gate failures or unresolved clarifications
|
|
||||||
@@ -1,257 +0,0 @@
|
|||||||
---
|
|
||||||
description: Create or update the feature specification from a natural language feature description.
|
|
||||||
handoffs:
|
|
||||||
- label: Build Technical Plan
|
|
||||||
agent: speckit.plan
|
|
||||||
prompt: Create a plan for the spec. I am building with...
|
|
||||||
- label: Clarify Spec Requirements
|
|
||||||
agent: speckit.clarify
|
|
||||||
prompt: Clarify specification requirements
|
|
||||||
send: true
|
|
||||||
---
|
|
||||||
|
|
||||||
## User Input
|
|
||||||
|
|
||||||
```text
|
|
||||||
$ARGUMENTS
|
|
||||||
```
|
|
||||||
|
|
||||||
You **MUST** consider the user input before proceeding (if not empty).
|
|
||||||
|
|
||||||
## Outline
|
|
||||||
|
|
||||||
The text the user typed after `/speckit.specify` in the triggering message **is** the feature description. Assume you always have it available in this conversation even if `$ARGUMENTS` appears literally below. Do not ask the user to repeat it unless they provided an empty command.
|
|
||||||
|
|
||||||
Given that feature description, do this:
|
|
||||||
|
|
||||||
1. **Generate a concise short name** (2-4 words) for the branch:
|
|
||||||
- Analyze the feature description and extract the most meaningful keywords
|
|
||||||
- Create a 2-4 word short name that captures the essence of the feature
|
|
||||||
- Use action-noun format when possible (e.g., "add-user-auth", "fix-payment-bug")
|
|
||||||
- Preserve technical terms and acronyms (OAuth2, API, JWT, etc.)
|
|
||||||
- Keep it concise but descriptive enough to understand the feature at a glance
|
|
||||||
- Examples:
|
|
||||||
- "I want to add user authentication" → "user-auth"
|
|
||||||
- "Implement OAuth2 integration for the API" → "oauth2-api-integration"
|
|
||||||
- "Create a dashboard for analytics" → "analytics-dashboard"
|
|
||||||
- "Fix payment processing timeout bug" → "fix-payment-timeout"
|
|
||||||
|
|
||||||
2. **Check for existing branches before creating new one**:
|
|
||||||
|
|
||||||
a. First, fetch all remote branches to ensure we have the latest information:
|
|
||||||
```bash
|
|
||||||
git fetch --all --prune
|
|
||||||
```
|
|
||||||
|
|
||||||
b. Find the highest feature number across all sources for the short-name:
|
|
||||||
- Remote branches: `git ls-remote --heads origin | grep -E 'refs/heads/[0-9]+-<short-name>$'`
|
|
||||||
- Local branches: `git branch | grep -E '^[* ]*[0-9]+-<short-name>$'`
|
|
||||||
- Specs directories: Check for directories matching `specs/[0-9]+-<short-name>`
|
|
||||||
|
|
||||||
c. Determine the next available number:
|
|
||||||
- Extract all numbers from all three sources
|
|
||||||
- Find the highest number N
|
|
||||||
- Use N+1 for the new branch number
|
|
||||||
|
|
||||||
d. Run the script `.specify/scripts/bash/create-new-feature.sh --json "$ARGUMENTS"` with the calculated number and short-name:
|
|
||||||
- Pass `--number N+1` and `--short-name "your-short-name"` along with the feature description
|
|
||||||
- Bash example: `.specify/scripts/bash/create-new-feature.sh --json "$ARGUMENTS" --json --number 5 --short-name "user-auth" "Add user authentication"`
|
|
||||||
- PowerShell example: `.specify/scripts/bash/create-new-feature.sh --json "$ARGUMENTS" -Json -Number 5 -ShortName "user-auth" "Add user authentication"`
|
|
||||||
|
|
||||||
**IMPORTANT**:
|
|
||||||
- Check all three sources (remote branches, local branches, specs directories) to find the highest number
|
|
||||||
- Only match branches/directories with the exact short-name pattern
|
|
||||||
- If no existing branches/directories found with this short-name, start with number 1
|
|
||||||
- You must only ever run this script once per feature
|
|
||||||
- The JSON is provided in the terminal as output - always refer to it to get the actual content you're looking for
|
|
||||||
- The JSON output will contain BRANCH_NAME and SPEC_FILE paths
|
|
||||||
- For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot")
|
|
||||||
|
|
||||||
3. Load `.specify/templates/spec-template.md` to understand required sections.
|
|
||||||
|
|
||||||
4. Follow this execution flow:
|
|
||||||
|
|
||||||
1. Parse user description from Input
|
|
||||||
If empty: ERROR "No feature description provided"
|
|
||||||
2. Extract key concepts from description
|
|
||||||
Identify: actors, actions, data, constraints
|
|
||||||
3. For unclear aspects:
|
|
||||||
- Make informed guesses based on context and industry standards
|
|
||||||
- Only mark with [NEEDS CLARIFICATION: specific question] if:
|
|
||||||
- The choice significantly impacts feature scope or user experience
|
|
||||||
- Multiple reasonable interpretations exist with different implications
|
|
||||||
- No reasonable default exists
|
|
||||||
- **LIMIT: Maximum 3 [NEEDS CLARIFICATION] markers total**
|
|
||||||
- Prioritize clarifications by impact: scope > security/privacy > user experience > technical details
|
|
||||||
4. Fill User Scenarios & Testing section
|
|
||||||
If no clear user flow: ERROR "Cannot determine user scenarios"
|
|
||||||
5. Generate Functional Requirements
|
|
||||||
Each requirement must be testable
|
|
||||||
Use reasonable defaults for unspecified details (document assumptions in Assumptions section)
|
|
||||||
6. Define Success Criteria
|
|
||||||
Create measurable, technology-agnostic outcomes
|
|
||||||
Include both quantitative metrics (time, performance, volume) and qualitative measures (user satisfaction, task completion)
|
|
||||||
Each criterion must be verifiable without implementation details
|
|
||||||
7. Identify Key Entities (if data involved)
|
|
||||||
8. Return: SUCCESS (spec ready for planning)
|
|
||||||
|
|
||||||
5. Write the specification to SPEC_FILE using the template structure, replacing placeholders with concrete details derived from the feature description (arguments) while preserving section order and headings.
|
|
||||||
|
|
||||||
6. **Specification Quality Validation**: After writing the initial spec, validate it against quality criteria:
|
|
||||||
|
|
||||||
a. **Create Spec Quality Checklist**: Generate a checklist file at `FEATURE_DIR/checklists/requirements.md` using the checklist template structure with these validation items:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
# Specification Quality Checklist: [FEATURE NAME]
|
|
||||||
|
|
||||||
**Purpose**: Validate specification completeness and quality before proceeding to planning
|
|
||||||
**Created**: [DATE]
|
|
||||||
**Feature**: [Link to spec.md]
|
|
||||||
|
|
||||||
## Content Quality
|
|
||||||
|
|
||||||
- [ ] No implementation details (languages, frameworks, APIs)
|
|
||||||
- [ ] Focused on user value and business needs
|
|
||||||
- [ ] Written for non-technical stakeholders
|
|
||||||
- [ ] All mandatory sections completed
|
|
||||||
|
|
||||||
## Requirement Completeness
|
|
||||||
|
|
||||||
- [ ] No [NEEDS CLARIFICATION] markers remain
|
|
||||||
- [ ] Requirements are testable and unambiguous
|
|
||||||
- [ ] Success criteria are measurable
|
|
||||||
- [ ] Success criteria are technology-agnostic (no implementation details)
|
|
||||||
- [ ] All acceptance scenarios are defined
|
|
||||||
- [ ] Edge cases are identified
|
|
||||||
- [ ] Scope is clearly bounded
|
|
||||||
- [ ] Dependencies and assumptions identified
|
|
||||||
|
|
||||||
## Feature Readiness
|
|
||||||
|
|
||||||
- [ ] All functional requirements have clear acceptance criteria
|
|
||||||
- [ ] User scenarios cover primary flows
|
|
||||||
- [ ] Feature meets measurable outcomes defined in Success Criteria
|
|
||||||
- [ ] No implementation details leak into specification
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan`
|
|
||||||
```
|
|
||||||
|
|
||||||
b. **Run Validation Check**: Review the spec against each checklist item:
|
|
||||||
- For each item, determine if it passes or fails
|
|
||||||
- Document specific issues found (quote relevant spec sections)
|
|
||||||
|
|
||||||
c. **Handle Validation Results**:
|
|
||||||
|
|
||||||
- **If all items pass**: Mark checklist complete and proceed to step 6
|
|
||||||
|
|
||||||
- **If items fail (excluding [NEEDS CLARIFICATION])**:
|
|
||||||
1. List the failing items and specific issues
|
|
||||||
2. Update the spec to address each issue
|
|
||||||
3. Re-run validation until all items pass (max 3 iterations)
|
|
||||||
4. If still failing after 3 iterations, document remaining issues in checklist notes and warn user
|
|
||||||
|
|
||||||
- **If [NEEDS CLARIFICATION] markers remain**:
|
|
||||||
1. Extract all [NEEDS CLARIFICATION: ...] markers from the spec
|
|
||||||
2. **LIMIT CHECK**: If more than 3 markers exist, keep only the 3 most critical (by scope/security/UX impact) and make informed guesses for the rest
|
|
||||||
3. For each clarification needed (max 3), present options to user in this format:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
## Question [N]: [Topic]
|
|
||||||
|
|
||||||
**Context**: [Quote relevant spec section]
|
|
||||||
|
|
||||||
**What we need to know**: [Specific question from NEEDS CLARIFICATION marker]
|
|
||||||
|
|
||||||
**Suggested Answers**:
|
|
||||||
|
|
||||||
| Option | Answer | Implications |
|
|
||||||
|--------|--------|--------------|
|
|
||||||
| A | [First suggested answer] | [What this means for the feature] |
|
|
||||||
| B | [Second suggested answer] | [What this means for the feature] |
|
|
||||||
| C | [Third suggested answer] | [What this means for the feature] |
|
|
||||||
| Custom | Provide your own answer | [Explain how to provide custom input] |
|
|
||||||
|
|
||||||
**Your choice**: _[Wait for user response]_
|
|
||||||
```
|
|
||||||
|
|
||||||
4. **CRITICAL - Table Formatting**: Ensure markdown tables are properly formatted:
|
|
||||||
- Use consistent spacing with pipes aligned
|
|
||||||
- Each cell should have spaces around content: `| Content |` not `|Content|`
|
|
||||||
- Header separator must have at least 3 dashes: `|--------|`
|
|
||||||
- Test that the table renders correctly in markdown preview
|
|
||||||
5. Number questions sequentially (Q1, Q2, Q3 - max 3 total)
|
|
||||||
6. Present all questions together before waiting for responses
|
|
||||||
7. Wait for user to respond with their choices for all questions (e.g., "Q1: A, Q2: Custom - [details], Q3: B")
|
|
||||||
8. Update the spec by replacing each [NEEDS CLARIFICATION] marker with the user's selected or provided answer
|
|
||||||
9. Re-run validation after all clarifications are resolved
|
|
||||||
|
|
||||||
d. **Update Checklist**: After each validation iteration, update the checklist file with current pass/fail status
|
|
||||||
|
|
||||||
7. Report completion with branch name, spec file path, checklist results, and readiness for the next phase (`/speckit.clarify` or `/speckit.plan`).
|
|
||||||
|
|
||||||
**NOTE:** The script creates and checks out the new branch and initializes the spec file before writing.
|
|
||||||
|
|
||||||
## General Guidelines
|
|
||||||
|
|
||||||
## Quick Guidelines
|
|
||||||
|
|
||||||
- Focus on **WHAT** users need and **WHY**.
|
|
||||||
- Avoid HOW to implement (no tech stack, APIs, code structure).
|
|
||||||
- Written for business stakeholders, not developers.
|
|
||||||
- DO NOT create any checklists that are embedded in the spec. That will be a separate command.
|
|
||||||
|
|
||||||
### Section Requirements
|
|
||||||
|
|
||||||
- **Mandatory sections**: Must be completed for every feature
|
|
||||||
- **Optional sections**: Include only when relevant to the feature
|
|
||||||
- When a section doesn't apply, remove it entirely (don't leave as "N/A")
|
|
||||||
|
|
||||||
### For AI Generation
|
|
||||||
|
|
||||||
When creating this spec from a user prompt:
|
|
||||||
|
|
||||||
1. **Make informed guesses**: Use context, industry standards, and common patterns to fill gaps
|
|
||||||
2. **Document assumptions**: Record reasonable defaults in the Assumptions section
|
|
||||||
3. **Limit clarifications**: Maximum 3 [NEEDS CLARIFICATION] markers - use only for critical decisions that:
|
|
||||||
- Significantly impact feature scope or user experience
|
|
||||||
- Have multiple reasonable interpretations with different implications
|
|
||||||
- Lack any reasonable default
|
|
||||||
4. **Prioritize clarifications**: scope > security/privacy > user experience > technical details
|
|
||||||
5. **Think like a tester**: Every vague requirement should fail the "testable and unambiguous" checklist item
|
|
||||||
6. **Common areas needing clarification** (only if no reasonable default exists):
|
|
||||||
- Feature scope and boundaries (include/exclude specific use cases)
|
|
||||||
- User types and permissions (if multiple conflicting interpretations possible)
|
|
||||||
- Security/compliance requirements (when legally/financially significant)
|
|
||||||
|
|
||||||
**Examples of reasonable defaults** (don't ask about these):
|
|
||||||
|
|
||||||
- Data retention: Industry-standard practices for the domain
|
|
||||||
- Performance targets: Standard web/mobile app expectations unless specified
|
|
||||||
- Error handling: User-friendly messages with appropriate fallbacks
|
|
||||||
- Authentication method: Standard session-based or OAuth2 for web apps
|
|
||||||
- Integration patterns: RESTful APIs unless specified otherwise
|
|
||||||
|
|
||||||
### Success Criteria Guidelines
|
|
||||||
|
|
||||||
Success criteria must be:
|
|
||||||
|
|
||||||
1. **Measurable**: Include specific metrics (time, percentage, count, rate)
|
|
||||||
2. **Technology-agnostic**: No mention of frameworks, languages, databases, or tools
|
|
||||||
3. **User-focused**: Describe outcomes from user/business perspective, not system internals
|
|
||||||
4. **Verifiable**: Can be tested/validated without knowing implementation details
|
|
||||||
|
|
||||||
**Good examples**:
|
|
||||||
|
|
||||||
- "Users can complete checkout in under 3 minutes"
|
|
||||||
- "System supports 10,000 concurrent users"
|
|
||||||
- "95% of searches return results in under 1 second"
|
|
||||||
- "Task completion rate improves by 40%"
|
|
||||||
|
|
||||||
**Bad examples** (implementation-focused):
|
|
||||||
|
|
||||||
- "API response time is under 200ms" (too technical, use "Users see results instantly")
|
|
||||||
- "Database can handle 1000 TPS" (implementation detail, use user-facing metric)
|
|
||||||
- "React components render efficiently" (framework-specific)
|
|
||||||
- "Redis cache hit rate above 80%" (technology-specific)
|
|
||||||
@@ -1,137 +0,0 @@
|
|||||||
---
|
|
||||||
description: Generate an actionable, dependency-ordered tasks.md for the feature based on available design artifacts.
|
|
||||||
handoffs:
|
|
||||||
- label: Analyze For Consistency
|
|
||||||
agent: speckit.analyze
|
|
||||||
prompt: Run a project analysis for consistency
|
|
||||||
send: true
|
|
||||||
- label: Implement Project
|
|
||||||
agent: speckit.implement
|
|
||||||
prompt: Start the implementation in phases
|
|
||||||
send: true
|
|
||||||
---
|
|
||||||
|
|
||||||
## User Input
|
|
||||||
|
|
||||||
```text
|
|
||||||
$ARGUMENTS
|
|
||||||
```
|
|
||||||
|
|
||||||
You **MUST** consider the user input before proceeding (if not empty).
|
|
||||||
|
|
||||||
## Outline
|
|
||||||
|
|
||||||
1. **Setup**: Run `.specify/scripts/bash/check-prerequisites.sh --json` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
|
||||||
|
|
||||||
2. **Load design documents**: Read from FEATURE_DIR:
|
|
||||||
- **Required**: plan.md (tech stack, libraries, structure), spec.md (user stories with priorities)
|
|
||||||
- **Optional**: data-model.md (entities), contracts/ (API endpoints), research.md (decisions), quickstart.md (test scenarios)
|
|
||||||
- Note: Not all projects have all documents. Generate tasks based on what's available.
|
|
||||||
|
|
||||||
3. **Execute task generation workflow**:
|
|
||||||
- Load plan.md and extract tech stack, libraries, project structure
|
|
||||||
- Load spec.md and extract user stories with their priorities (P1, P2, P3, etc.)
|
|
||||||
- If data-model.md exists: Extract entities and map to user stories
|
|
||||||
- If contracts/ exists: Map endpoints to user stories
|
|
||||||
- If research.md exists: Extract decisions for setup tasks
|
|
||||||
- Generate tasks organized by user story (see Task Generation Rules below)
|
|
||||||
- Generate dependency graph showing user story completion order
|
|
||||||
- Create parallel execution examples per user story
|
|
||||||
- Validate task completeness (each user story has all needed tasks, independently testable)
|
|
||||||
|
|
||||||
4. **Generate tasks.md**: Use `.specify.specify/templates/tasks-template.md` as structure, fill with:
|
|
||||||
- Correct feature name from plan.md
|
|
||||||
- Phase 1: Setup tasks (project initialization)
|
|
||||||
- Phase 2: Foundational tasks (blocking prerequisites for all user stories)
|
|
||||||
- Phase 3+: One phase per user story (in priority order from spec.md)
|
|
||||||
- Each phase includes: story goal, independent test criteria, tests (if requested), implementation tasks
|
|
||||||
- Final Phase: Polish & cross-cutting concerns
|
|
||||||
- All tasks must follow the strict checklist format (see Task Generation Rules below)
|
|
||||||
- Clear file paths for each task
|
|
||||||
- Dependencies section showing story completion order
|
|
||||||
- Parallel execution examples per story
|
|
||||||
- Implementation strategy section (MVP first, incremental delivery)
|
|
||||||
|
|
||||||
5. **Report**: Output path to generated tasks.md and summary:
|
|
||||||
- Total task count
|
|
||||||
- Task count per user story
|
|
||||||
- Parallel opportunities identified
|
|
||||||
- Independent test criteria for each story
|
|
||||||
- Suggested MVP scope (typically just User Story 1)
|
|
||||||
- Format validation: Confirm ALL tasks follow the checklist format (checkbox, ID, labels, file paths)
|
|
||||||
|
|
||||||
Context for task generation: $ARGUMENTS
|
|
||||||
|
|
||||||
The tasks.md should be immediately executable - each task must be specific enough that an LLM can complete it without additional context.
|
|
||||||
|
|
||||||
## Task Generation Rules
|
|
||||||
|
|
||||||
**CRITICAL**: Tasks MUST be organized by user story to enable independent implementation and testing.
|
|
||||||
|
|
||||||
**Tests are OPTIONAL**: Only generate test tasks if explicitly requested in the feature specification or if user requests TDD approach.
|
|
||||||
|
|
||||||
### Checklist Format (REQUIRED)
|
|
||||||
|
|
||||||
Every task MUST strictly follow this format:
|
|
||||||
|
|
||||||
```text
|
|
||||||
- [ ] [TaskID] [P?] [Story?] Description with file path
|
|
||||||
```
|
|
||||||
|
|
||||||
**Format Components**:
|
|
||||||
|
|
||||||
1. **Checkbox**: ALWAYS start with `- [ ]` (markdown checkbox)
|
|
||||||
2. **Task ID**: Sequential number (T001, T002, T003...) in execution order
|
|
||||||
3. **[P] marker**: Include ONLY if task is parallelizable (different files, no dependencies on incomplete tasks)
|
|
||||||
4. **[Story] label**: REQUIRED for user story phase tasks only
|
|
||||||
- Format: [US1], [US2], [US3], etc. (maps to user stories from spec.md)
|
|
||||||
- Setup phase: NO story label
|
|
||||||
- Foundational phase: NO story label
|
|
||||||
- User Story phases: MUST have story label
|
|
||||||
- Polish phase: NO story label
|
|
||||||
5. **Description**: Clear action with exact file path
|
|
||||||
|
|
||||||
**Examples**:
|
|
||||||
|
|
||||||
- ✅ CORRECT: `- [ ] T001 Create project structure per implementation plan`
|
|
||||||
- ✅ CORRECT: `- [ ] T005 [P] Implement authentication middleware in src/middleware/auth.py`
|
|
||||||
- ✅ CORRECT: `- [ ] T012 [P] [US1] Create User model in src/models/user.py`
|
|
||||||
- ✅ CORRECT: `- [ ] T014 [US1] Implement UserService in src/services/user_service.py`
|
|
||||||
- ❌ WRONG: `- [ ] Create User model` (missing ID and Story label)
|
|
||||||
- ❌ WRONG: `T001 [US1] Create model` (missing checkbox)
|
|
||||||
- ❌ WRONG: `- [ ] [US1] Create User model` (missing Task ID)
|
|
||||||
- ❌ WRONG: `- [ ] T001 [US1] Create model` (missing file path)
|
|
||||||
|
|
||||||
### Task Organization
|
|
||||||
|
|
||||||
1. **From User Stories (spec.md)** - PRIMARY ORGANIZATION:
|
|
||||||
- Each user story (P1, P2, P3...) gets its own phase
|
|
||||||
- Map all related components to their story:
|
|
||||||
- Models needed for that story
|
|
||||||
- Services needed for that story
|
|
||||||
- Endpoints/UI needed for that story
|
|
||||||
- If tests requested: Tests specific to that story
|
|
||||||
- Mark story dependencies (most stories should be independent)
|
|
||||||
|
|
||||||
2. **From Contracts**:
|
|
||||||
- Map each contract/endpoint → to the user story it serves
|
|
||||||
- If tests requested: Each contract → contract test task [P] before implementation in that story's phase
|
|
||||||
|
|
||||||
3. **From Data Model**:
|
|
||||||
- Map each entity to the user story(ies) that need it
|
|
||||||
- If entity serves multiple stories: Put in earliest story or Setup phase
|
|
||||||
- Relationships → service layer tasks in appropriate story phase
|
|
||||||
|
|
||||||
4. **From Setup/Infrastructure**:
|
|
||||||
- Shared infrastructure → Setup phase (Phase 1)
|
|
||||||
- Foundational/blocking tasks → Foundational phase (Phase 2)
|
|
||||||
- Story-specific setup → within that story's phase
|
|
||||||
|
|
||||||
### Phase Structure
|
|
||||||
|
|
||||||
- **Phase 1**: Setup (project initialization)
|
|
||||||
- **Phase 2**: Foundational (blocking prerequisites - MUST complete before user stories)
|
|
||||||
- **Phase 3+**: User Stories in priority order (P1, P2, P3...)
|
|
||||||
- Within each story: Tests (if requested) → Models → Services → Endpoints → Integration
|
|
||||||
- Each phase should be a complete, independently testable increment
|
|
||||||
- **Final Phase**: Polish & Cross-Cutting Concerns
|
|
||||||
@@ -1,28 +0,0 @@
|
|||||||
---
|
|
||||||
description: Convert existing tasks into actionable, dependency-ordered GitHub issues for the feature based on available design artifacts.
|
|
||||||
tools: ['github/github-mcp-server/issue_write']
|
|
||||||
---
|
|
||||||
|
|
||||||
## User Input
|
|
||||||
|
|
||||||
```text
|
|
||||||
$ARGUMENTS
|
|
||||||
```
|
|
||||||
|
|
||||||
You **MUST** consider the user input before proceeding (if not empty).
|
|
||||||
|
|
||||||
## Outline
|
|
||||||
|
|
||||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute. For single quotes in args like "I'm Groot", use escape syntax: e.g 'I'\''m Groot' (or double-quote if possible: "I'm Groot").
|
|
||||||
1. From the executed script, extract the path to **tasks**.
|
|
||||||
1. Get the Git remote by running:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git config --get remote.origin.url
|
|
||||||
```
|
|
||||||
|
|
||||||
**ONLY PROCEED TO NEXT STEPS IF THE REMOTE IS A GITHUB URL**
|
|
||||||
|
|
||||||
1. For each task in the list, use the GitHub MCP server to create a new issue in the repository that is representative of the Git remote.
|
|
||||||
|
|
||||||
**UNDER NO CIRCUMSTANCES EVER CREATE ISSUES IN REPOSITORIES THAT DO NOT MATCH THE REMOTE URL**
|
|
||||||
@@ -1,6 +0,0 @@
|
|||||||
[codespell]
|
|
||||||
skip = .git,*.pdf,*.svg,timeline-expected.html,*.fq,*.min.js,ScriptDslTest.groovy
|
|
||||||
# some cases where we need to catch using regex
|
|
||||||
ignore-regex = \bhel\*|fo\\
|
|
||||||
# some variables, names, etc to ignore
|
|
||||||
ignore-words-list = splitted,ois,tre,marge,smoot,afile,bams,bais,pre-pending,re-use
|
|
||||||
@@ -1,8 +0,0 @@
|
|||||||
[*]
|
|
||||||
charset = utf-8
|
|
||||||
indent_size = 4
|
|
||||||
indent_style = space
|
|
||||||
tab_width = 4
|
|
||||||
end_of_line = lf
|
|
||||||
insert_final_newline = true
|
|
||||||
trim_trailing_whitespace = true
|
|
||||||
7
nextflow/.github/CODEOWNERS
vendored
7
nextflow/.github/CODEOWNERS
vendored
@@ -1,7 +0,0 @@
|
|||||||
# DOCUMENTATION -------------------------------------------------------
|
|
||||||
# Docs folder ownership
|
|
||||||
/docs/ @nextflow-io/docs
|
|
||||||
|
|
||||||
# COMPILER ------------------------------------------------------------
|
|
||||||
# nf-lang module
|
|
||||||
/modules/nf-lang/ @nextflow-io/lang
|
|
||||||
31
nextflow/.github/ISSUE_TEMPLATE/bug_report.md
vendored
31
nextflow/.github/ISSUE_TEMPLATE/bug_report.md
vendored
@@ -1,31 +0,0 @@
|
|||||||
---
|
|
||||||
name: Bug report
|
|
||||||
about: Report a bug to help us improve
|
|
||||||
---
|
|
||||||
|
|
||||||
## Bug report
|
|
||||||
|
|
||||||
(Please follow this template by replacing the text between parentheses with the requested information)
|
|
||||||
|
|
||||||
### Expected behavior and actual behavior
|
|
||||||
|
|
||||||
(Give a brief description of the expected behavior and actual behavior)
|
|
||||||
|
|
||||||
### Steps to reproduce the problem
|
|
||||||
|
|
||||||
(Provide a test case that reproduces the problem either with a self-contained script or GitHub repository)
|
|
||||||
|
|
||||||
### Program output
|
|
||||||
|
|
||||||
(Copy and paste the output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the `.nextflow.log` file.)
|
|
||||||
|
|
||||||
### Environment
|
|
||||||
|
|
||||||
* Nextflow version: [?]
|
|
||||||
* Java version: [?]
|
|
||||||
* Operating system: [macOS, Linux, etc]
|
|
||||||
* Bash version: (use the command `$SHELL --version`)
|
|
||||||
|
|
||||||
### Additional context
|
|
||||||
|
|
||||||
(Add any other context about the problem here)
|
|
||||||
11
nextflow/.github/ISSUE_TEMPLATE/config.yml
vendored
11
nextflow/.github/ISSUE_TEMPLATE/config.yml
vendored
@@ -1,11 +0,0 @@
|
|||||||
blank_issues_enabled: false
|
|
||||||
contact_links:
|
|
||||||
- name: General question
|
|
||||||
url: https://community.seqera.io/nextflow
|
|
||||||
about: Ask for help with Nextflow language and usage
|
|
||||||
- name: Nextflow official website
|
|
||||||
url: https://nextflow.io/
|
|
||||||
about: Documentation and tutorials
|
|
||||||
- name: Nextflow training
|
|
||||||
url: https://training.nextflow.io/
|
|
||||||
about: Learn Nextflow with hands-on tutorials
|
|
||||||
16
nextflow/.github/ISSUE_TEMPLATE/new_feature.md
vendored
16
nextflow/.github/ISSUE_TEMPLATE/new_feature.md
vendored
@@ -1,16 +0,0 @@
|
|||||||
---
|
|
||||||
name: New feature
|
|
||||||
about: Propose a new feature or enhancement
|
|
||||||
---
|
|
||||||
|
|
||||||
## New feature
|
|
||||||
|
|
||||||
(Hi! Thanks for using Nextflow and for proposing a new feature or enhancement. Please replace this text with a brief description of your proposal.)
|
|
||||||
|
|
||||||
## Use case
|
|
||||||
|
|
||||||
(What's the main use case and deployment scenario addressed by this proposal)
|
|
||||||
|
|
||||||
## Suggested implementation
|
|
||||||
|
|
||||||
(Highlight the main building blocks of a possible implementation and/or related components)
|
|
||||||
@@ -1,7 +0,0 @@
|
|||||||
Hi! Thanks for contributing to Nextflow.
|
|
||||||
|
|
||||||
When submitting a Pull Request, please sign-off the DCO [1] to certify that you are the author of the contribution and you adhere to Nextflow's open source license [2] by adding a `Signed-off-by` line to the contribution commit message. See [3] for more details.
|
|
||||||
|
|
||||||
1. https://developercertificate.org/
|
|
||||||
2. https://github.com/nextflow-io/nextflow/blob/master/COPYING
|
|
||||||
3. https://github.com/apps/dco
|
|
||||||
15
nextflow/.github/workflows/README.md
vendored
15
nextflow/.github/workflows/README.md
vendored
@@ -1,15 +0,0 @@
|
|||||||
# Action
|
|
||||||
|
|
||||||
## Syntax
|
|
||||||
|
|
||||||
https://help.github.com/en/articles/workflow-syntax-for-github-actions
|
|
||||||
https://help.github.com/en/articles/contexts-and-expression-syntax-for-github-actions
|
|
||||||
https://help.github.com/en/articles/virtual-environments-for-github-actions#environment-variables
|
|
||||||
https://help.github.com/en/articles/configuring-docker-for-use-with-github-package-registry
|
|
||||||
https://help.github.com/en/articles/virtual-environments-for-github-actions#creating-and-using-secrets-encrypted-variables
|
|
||||||
|
|
||||||
## Java
|
|
||||||
|
|
||||||
Java VMs has to match the ones at this link https://static.azul.com/zulu/bin
|
|
||||||
|
|
||||||
Check the name *-jdk(x.y.z)
|
|
||||||
296
nextflow/.github/workflows/build.yml
vendored
296
nextflow/.github/workflows/build.yml
vendored
@@ -1,296 +0,0 @@
|
|||||||
name: Nextflow CI
|
|
||||||
# read more here: https://help.github.com/en/articles/workflow-syntax-for-github-actions#on
|
|
||||||
|
|
||||||
# Note: We don't use the `on: path` option for docs,
|
|
||||||
# because the Build steps are *required* tests.
|
|
||||||
# Instead, we trigger + skip the tests if the only changes
|
|
||||||
# are in the docs folder. GitHub treats this as passing.
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches:
|
|
||||||
- 'master'
|
|
||||||
- 'test*'
|
|
||||||
- 'dev*'
|
|
||||||
- 'STABLE-*'
|
|
||||||
pull_request:
|
|
||||||
types: [opened, reopened, synchronize]
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
build:
|
|
||||||
name: Build
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
timeout-minutes: 100
|
|
||||||
strategy:
|
|
||||||
fail-fast: false
|
|
||||||
matrix:
|
|
||||||
java_version: [17, 25]
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
with:
|
|
||||||
fetch-depth: 1
|
|
||||||
submodules: true
|
|
||||||
|
|
||||||
- name: Get the commit message
|
|
||||||
id: get_commit_message
|
|
||||||
run: |
|
|
||||||
if [ "${{ github.event_name }}" = "pull_request" ]; then
|
|
||||||
echo "GitHub event=pull_request"
|
|
||||||
COMMIT_SHA="${{ github.event.pull_request.head.sha }}"
|
|
||||||
COMMIT_MESSAGE="$(curl -s \
|
|
||||||
-H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
|
|
||||||
https://api.github.com/repos/${{ github.repository }}/commits/$COMMIT_SHA | jq -r '.commit.message' | head -n 1)"
|
|
||||||
echo "Commit message=$(printf "%s" "$COMMIT_MESSAGE")"
|
|
||||||
echo "commit_message=$(printf "%s" "$COMMIT_MESSAGE")" >> $GITHUB_OUTPUT
|
|
||||||
else
|
|
||||||
echo "GitHub event=${{ github.event_name }}"
|
|
||||||
# Extract only the first line of the commit message
|
|
||||||
COMMIT_MESSAGE="$(git log -1 --pretty=format:'%s')"
|
|
||||||
echo "Commit message=$(printf "%s" "$COMMIT_MESSAGE")"
|
|
||||||
echo "commit_message=$(printf "%s" "$COMMIT_MESSAGE")" >> $GITHUB_OUTPUT
|
|
||||||
fi
|
|
||||||
|
|
||||||
- name: Setup env
|
|
||||||
run: |
|
|
||||||
rm -f $HOME/.gitconfig;
|
|
||||||
mkdir -p "$HOME/.nextflow";
|
|
||||||
echo "providers.github.auth='$NXF_GITHUB_ACCESS_TOKEN'" > "$HOME/.nextflow/scm"
|
|
||||||
env:
|
|
||||||
NXF_GITHUB_ACCESS_TOKEN: ${{ secrets.NXF_GITHUB_ACCESS_TOKEN }}
|
|
||||||
|
|
||||||
- name: Setup Java ${{ matrix.java_version }}
|
|
||||||
uses: actions/setup-java@v4
|
|
||||||
with:
|
|
||||||
java-version: ${{matrix.java_version}}
|
|
||||||
distribution: 'temurin'
|
|
||||||
architecture: x64
|
|
||||||
cache: gradle
|
|
||||||
|
|
||||||
- name: Compile
|
|
||||||
run: make assemble
|
|
||||||
|
|
||||||
- name: Test
|
|
||||||
run: |
|
|
||||||
env | sort
|
|
||||||
# configure test env
|
|
||||||
if [[ "$GOOGLE_SECRET" ]]; then
|
|
||||||
echo $GOOGLE_SECRET | base64 -d > $PWD/google_credentials.json
|
|
||||||
export GOOGLE_APPLICATION_CREDENTIALS=$PWD/google_credentials.json
|
|
||||||
fi
|
|
||||||
# run tests
|
|
||||||
make test
|
|
||||||
env:
|
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
AWS_S3FS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_S3FS_SECRET_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
NXF_BITBUCKET_ACCESS_TOKEN: ${{ secrets.NXF_BITBUCKET_ACCESS_TOKEN }}
|
|
||||||
NXF_GITHUB_ACCESS_TOKEN: ${{ secrets.NXF_GITHUB_ACCESS_TOKEN }}
|
|
||||||
NXF_GITLAB_ACCESS_TOKEN: ${{ secrets.NXF_GITLAB_ACCESS_TOKEN }}
|
|
||||||
NXF_AZURE_REPOS_TOKEN: ${{ secrets.NXF_AZURE_REPOS_TOKEN }}
|
|
||||||
GOOGLE_SECRET: ${{ secrets.GOOGLE_SECRET }}
|
|
||||||
AZURE_STORAGE_ACCOUNT_NAME: nfazurestore
|
|
||||||
AZURE_STORAGE_ACCOUNT_KEY: ${{ secrets.AZURE_STORAGE_ACCOUNT_KEY }}
|
|
||||||
AZURE_BATCH_ACCOUNT_NAME: nfbatchtest
|
|
||||||
AZURE_BATCH_ACCOUNT_KEY: ${{ secrets.AZURE_BATCH_ACCOUNT_KEY }}
|
|
||||||
|
|
||||||
- name: Publish tests report
|
|
||||||
if: always()
|
|
||||||
uses: actions/upload-artifact@v4
|
|
||||||
with:
|
|
||||||
name: report-unit-tests-jdk-${{ matrix.java_version }}
|
|
||||||
path: |
|
|
||||||
**/build/reports/tests/test
|
|
||||||
|
|
||||||
outputs:
|
|
||||||
commit_message: ${{ steps.get_commit_message.outputs.commit_message }}
|
|
||||||
|
|
||||||
test:
|
|
||||||
if: ${{ !contains(needs.build.outputs.commit_message, '[ci fast]') }}
|
|
||||||
needs: build
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
timeout-minutes: 90
|
|
||||||
strategy:
|
|
||||||
fail-fast: false
|
|
||||||
matrix:
|
|
||||||
java_version: [17, 25]
|
|
||||||
test_mode: ["test_integration", "test_parser_v2", "test_docs", "test_aws", "test_azure", "test_google", "test_wave"]
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
with:
|
|
||||||
fetch-depth: 1
|
|
||||||
submodules: true
|
|
||||||
|
|
||||||
- name: Setup env
|
|
||||||
run: |
|
|
||||||
rm -f $HOME/.gitconfig;
|
|
||||||
mkdir -p "$HOME/.nextflow";
|
|
||||||
echo "providers.github.auth='$NXF_GITHUB_ACCESS_TOKEN'" > "$HOME/.nextflow/scm"
|
|
||||||
env:
|
|
||||||
NXF_GITHUB_ACCESS_TOKEN: ${{ secrets.NXF_GITHUB_ACCESS_TOKEN }}
|
|
||||||
|
|
||||||
- name: Setup Java ${{ matrix.java_version }}
|
|
||||||
uses: actions/setup-java@v4
|
|
||||||
with:
|
|
||||||
java-version: ${{matrix.java_version}}
|
|
||||||
distribution: 'temurin'
|
|
||||||
architecture: x64
|
|
||||||
cache: gradle
|
|
||||||
|
|
||||||
- name: Run tests
|
|
||||||
run: |
|
|
||||||
env | sort
|
|
||||||
# configure test env
|
|
||||||
if [[ "$GOOGLE_SECRET" ]]; then
|
|
||||||
echo $GOOGLE_SECRET | base64 -d > $PWD/google_credentials.json
|
|
||||||
export GOOGLE_APPLICATION_CREDENTIALS=$PWD/google_credentials.json
|
|
||||||
fi
|
|
||||||
cat $HOME/.nextflow/scm
|
|
||||||
make clean assemble install
|
|
||||||
bash test-ci.sh
|
|
||||||
env:
|
|
||||||
TEST_JDK: ${{ matrix.java_version }}
|
|
||||||
TEST_MODE: ${{ matrix.test_mode }}
|
|
||||||
GRADLE_OPTS: '-Dorg.gradle.daemon=false'
|
|
||||||
TOWER_ACCESS_TOKEN: ${{ secrets.TOWER_ACCESS_TOKEN }}
|
|
||||||
AWS_DEFAULT_REGION: eu-west-1
|
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
NXF_BITBUCKET_ACCESS_TOKEN: ${{ secrets.NXF_BITBUCKET_ACCESS_TOKEN }}
|
|
||||||
NXF_GITHUB_ACCESS_TOKEN: ${{ secrets.NXF_GITHUB_ACCESS_TOKEN }}
|
|
||||||
NXF_GITLAB_ACCESS_TOKEN: ${{ secrets.NXF_GITLAB_ACCESS_TOKEN }}
|
|
||||||
NXF_AZURE_REPOS_TOKEN: ${{ secrets.NXF_AZURE_REPOS_TOKEN }}
|
|
||||||
GOOGLE_SECRET: ${{ secrets.GOOGLE_SECRET }}
|
|
||||||
AZURE_STORAGE_ACCOUNT_NAME: nfazurestore
|
|
||||||
AZURE_STORAGE_ACCOUNT_KEY: ${{ secrets.AZURE_STORAGE_ACCOUNT_KEY }}
|
|
||||||
AZURE_BATCH_ACCOUNT_NAME: nfbatchtest
|
|
||||||
AZURE_BATCH_ACCOUNT_KEY: ${{ secrets.AZURE_BATCH_ACCOUNT_KEY }}
|
|
||||||
|
|
||||||
- name: Tar integration tests
|
|
||||||
if: always()
|
|
||||||
run: |
|
|
||||||
tar -cvf integration-tests.tar.gz tests/checks
|
|
||||||
tar -cvf validation-tests.tar.gz validation
|
|
||||||
|
|
||||||
- name: Publish tests report
|
|
||||||
uses: actions/upload-artifact@v4
|
|
||||||
if: always()
|
|
||||||
with:
|
|
||||||
name: report-${{ matrix.test_mode }}-jdk-${{ matrix.java_version }}
|
|
||||||
path: |
|
|
||||||
validation-tests.tar.gz
|
|
||||||
integration-tests.tar.gz
|
|
||||||
|
|
||||||
test-e2e:
|
|
||||||
if: ${{ contains(needs.build.outputs.commit_message,'[e2e stage]') || contains(needs.build.outputs.commit_message,'[e2e prod]') }}
|
|
||||||
needs: build
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
timeout-minutes: 10
|
|
||||||
permissions:
|
|
||||||
actions: write # Allow writing to actions
|
|
||||||
contents: write # Allow writing to repository contents
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
with:
|
|
||||||
fetch-depth: 1
|
|
||||||
submodules: true
|
|
||||||
|
|
||||||
- name: Setup Java 17
|
|
||||||
uses: actions/setup-java@v4
|
|
||||||
with:
|
|
||||||
java-version: 17
|
|
||||||
distribution: 'temurin'
|
|
||||||
architecture: x64
|
|
||||||
cache: gradle
|
|
||||||
|
|
||||||
- name: Setup env
|
|
||||||
run: |
|
|
||||||
wget -q -O wave https://github.com/seqeralabs/wave-cli/releases/download/v1.4.1/wave-1.4.1-linux-x86_64
|
|
||||||
chmod +x wave
|
|
||||||
mv wave /usr/local/bin/
|
|
||||||
echo "COMMIT_MESSAGE=\"${{ needs.build.outputs.commit_message }}\"" >> $GITHUB_ENV
|
|
||||||
|
|
||||||
- name : Docker Login to Seqera public CR
|
|
||||||
uses : docker/login-action@v3
|
|
||||||
with :
|
|
||||||
registry : "public.cr.seqera.io"
|
|
||||||
username : "public-cr-admin"
|
|
||||||
password : ${{ secrets.SEQERA_PUBLIC_CR_PASSWORD }}
|
|
||||||
|
|
||||||
- name: Launch tests
|
|
||||||
run: |
|
|
||||||
cd test-e2e
|
|
||||||
bash run.sh
|
|
||||||
env:
|
|
||||||
GITHUB_TOKEN: ${{ secrets.AUTOMATION_GITHUB_TOKEN }}
|
|
||||||
GRADLE_OPTS: '-Dorg.gradle.daemon=false'
|
|
||||||
|
|
||||||
release:
|
|
||||||
if: ${{ always() && contains(needs.build.outputs.commit_message, '[release]') && needs.build.result == 'success' && (needs.test.result == 'success' || needs.test.result == 'skipped') }}
|
|
||||||
needs: [build, test]
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
timeout-minutes: 10
|
|
||||||
permissions:
|
|
||||||
actions: write
|
|
||||||
contents: write
|
|
||||||
packages: write
|
|
||||||
pull-requests: write
|
|
||||||
issues: write
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
with:
|
|
||||||
fetch-depth: 0
|
|
||||||
submodules: true
|
|
||||||
|
|
||||||
- name: Setup Java 17
|
|
||||||
uses: actions/setup-java@v4
|
|
||||||
with:
|
|
||||||
java-version: 17
|
|
||||||
distribution: 'temurin'
|
|
||||||
architecture: x64
|
|
||||||
cache: gradle
|
|
||||||
|
|
||||||
- name: Configure Git
|
|
||||||
run: |
|
|
||||||
git config --global user.name "${{ github.event.pusher.name || github.actor }}"
|
|
||||||
git config --global user.email "${{ github.event.pusher.email || format('{0}@users.noreply.github.com', github.actor) }}"
|
|
||||||
|
|
||||||
- name: Docker Login to Docker Hub
|
|
||||||
uses: docker/login-action@v3
|
|
||||||
with:
|
|
||||||
username: ${{ vars.DOCKERHUB_USERNAME }}
|
|
||||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
|
||||||
|
|
||||||
- name: Docker Login to Seqera public CR
|
|
||||||
uses: docker/login-action@v3
|
|
||||||
with:
|
|
||||||
registry: "public.cr.seqera.io"
|
|
||||||
username: ${{ vars.SEQERA_PUBLIC_CR_USERNAME }}
|
|
||||||
password: ${{ secrets.SEQERA_PUBLIC_CR_PASSWORD }}
|
|
||||||
|
|
||||||
- name: Run release
|
|
||||||
run: |
|
|
||||||
echo "Starting release process..."
|
|
||||||
echo "npr.apiUrl=$NPR_API_URL" >> gradle.properties
|
|
||||||
echo "npr.apiKey=$NPR_API_KEY" >> gradle.properties
|
|
||||||
bash release.sh
|
|
||||||
env:
|
|
||||||
GRADLE_OPTS: '-Dorg.gradle.daemon=false'
|
|
||||||
AWS_JAVA_V1_DISABLE_DEPRECATION_ANNOUNCEMENT: 'true'
|
|
||||||
# credentials to pubslish nextflow assets
|
|
||||||
NXF_AWS_ACCESS: ${{ vars.NXF_AWS_ACCESS }}
|
|
||||||
NXF_AWS_SECRET: ${{ secrets.NXF_AWS_SECRET }}
|
|
||||||
# credentials to publish maven libraries
|
|
||||||
AWS_ACCESS_KEY_ID: ${{ vars.SEQERA_MAVEN_ACCESS_KEY }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.SEQERA_MAVEN_SECRET_KEY }}
|
|
||||||
# plugin registry
|
|
||||||
NPR_API_URL: ${{ vars.NPR_API_URL }}
|
|
||||||
NPR_API_KEY: ${{ secrets.NPR_API_KEY }}
|
|
||||||
# GitHub secrets
|
|
||||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
||||||
19
nextflow/.github/workflows/cffconvert.yml
vendored
19
nextflow/.github/workflows/cffconvert.yml
vendored
@@ -1,19 +0,0 @@
|
|||||||
name: cffconvert
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
paths:
|
|
||||||
- CITATION.cff
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
validate:
|
|
||||||
name: "validate"
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- name: Check out a copy of the repository
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- name: Check whether the citation metadata from CITATION.cff is valid
|
|
||||||
uses: citation-file-format/cffconvert-github-action@2.0.0
|
|
||||||
with:
|
|
||||||
args: "--validate"
|
|
||||||
36
nextflow/.github/workflows/claude.yml
vendored
36
nextflow/.github/workflows/claude.yml
vendored
@@ -1,36 +0,0 @@
|
|||||||
name: Claude PR Assistant
|
|
||||||
|
|
||||||
on:
|
|
||||||
issue_comment:
|
|
||||||
types: [created]
|
|
||||||
pull_request_review_comment:
|
|
||||||
types: [created]
|
|
||||||
issues:
|
|
||||||
types: [opened, assigned]
|
|
||||||
pull_request_review:
|
|
||||||
types: [submitted]
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
claude-code-action:
|
|
||||||
if: |
|
|
||||||
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
|
|
||||||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
|
|
||||||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
|
|
||||||
(github.event_name == 'issues' && contains(github.event.issue.body, '@claude'))
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
pull-requests: read
|
|
||||||
issues: read
|
|
||||||
id-token: write
|
|
||||||
steps:
|
|
||||||
- name: Checkout repository
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
with:
|
|
||||||
fetch-depth: 1
|
|
||||||
|
|
||||||
- name: Run Claude PR Action
|
|
||||||
uses: anthropics/claude-code-action@beta
|
|
||||||
with:
|
|
||||||
anthropic_api_key: ${{ secrets.NEXTFLOW_ANTHROPIC_API_KEY }}
|
|
||||||
timeout_minutes: "60"
|
|
||||||
22
nextflow/.github/workflows/docs.yml
vendored
22
nextflow/.github/workflows/docs.yml
vendored
@@ -1,22 +0,0 @@
|
|||||||
name: Docs CI
|
|
||||||
on:
|
|
||||||
pull_request:
|
|
||||||
types: [opened, reopened, synchronize]
|
|
||||||
paths:
|
|
||||||
- 'docs/**'
|
|
||||||
workflow_dispatch:
|
|
||||||
jobs:
|
|
||||||
docs-build:
|
|
||||||
name: Build
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v4
|
|
||||||
- uses: actions/setup-python@v4
|
|
||||||
with:
|
|
||||||
python-version: '3.9'
|
|
||||||
|
|
||||||
- name: Test docs build
|
|
||||||
run: |
|
|
||||||
cd docs/
|
|
||||||
pip install -r requirements.txt
|
|
||||||
make clean html
|
|
||||||
21
nextflow/.github/workflows/stale.yml
vendored
21
nextflow/.github/workflows/stale.yml
vendored
@@ -1,21 +0,0 @@
|
|||||||
name: 'Mark stale issues and PRs'
|
|
||||||
on:
|
|
||||||
schedule:
|
|
||||||
- cron: '30 1 * * *'
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
stale:
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
- uses: actions/stale@v9
|
|
||||||
with:
|
|
||||||
days-before-stale: 180
|
|
||||||
days-before-close: -1
|
|
||||||
stale-issue-label: stale
|
|
||||||
stale-issue-message: ''
|
|
||||||
stale-pr-label: stale
|
|
||||||
stale-pr-message: ''
|
|
||||||
exempt-issue-labels: bug,planned,security
|
|
||||||
exempt-all-milestones: true
|
|
||||||
exempt-all-assignees: true
|
|
||||||
48
nextflow/.gitignore
vendored
48
nextflow/.gitignore
vendored
@@ -1,48 +0,0 @@
|
|||||||
.idea/**
|
|
||||||
.gradle
|
|
||||||
.*.log*
|
|
||||||
.cache/*
|
|
||||||
.vagrant/*
|
|
||||||
.cache/*
|
|
||||||
.nextflow*
|
|
||||||
.node-nextflow*
|
|
||||||
.devcontainer
|
|
||||||
.vscode/*
|
|
||||||
.lineage/
|
|
||||||
*/*/bin/*
|
|
||||||
**/build/**
|
|
||||||
build/**
|
|
||||||
modules/**/build/
|
|
||||||
modules/**/*.iml
|
|
||||||
modules/nextflow/work/
|
|
||||||
**/out/*
|
|
||||||
tmp/*
|
|
||||||
work/*
|
|
||||||
nextflow.eml
|
|
||||||
nextflow.iml
|
|
||||||
Vagrantfile
|
|
||||||
resources/*
|
|
||||||
test/*
|
|
||||||
tools/*
|
|
||||||
tests_other
|
|
||||||
other
|
|
||||||
docs/_build/
|
|
||||||
./*.txt
|
|
||||||
errors/
|
|
||||||
misc/
|
|
||||||
cloud/my-*
|
|
||||||
docker-test
|
|
||||||
docker/dist
|
|
||||||
docker/nextflow
|
|
||||||
temp
|
|
||||||
.dockerignore
|
|
||||||
.launch.classpath
|
|
||||||
plugins-prod
|
|
||||||
/minio
|
|
||||||
/sandbox
|
|
||||||
/wave-tests
|
|
||||||
/test-sched
|
|
||||||
/test-module
|
|
||||||
/results
|
|
||||||
/x/*
|
|
||||||
mise.toml
|
|
||||||
@@ -1,198 +0,0 @@
|
|||||||
# Nextflow Development Constitution
|
|
||||||
|
|
||||||
<!--
|
|
||||||
SYNC IMPACT REPORT
|
|
||||||
==================
|
|
||||||
Version Change: INITIAL → 1.0.0 (Initial constitution)
|
|
||||||
Modified Principles: N/A (new constitution)
|
|
||||||
Added Sections:
|
|
||||||
- All principles (I-VII)
|
|
||||||
- Development Workflow
|
|
||||||
- Quality Standards
|
|
||||||
- Governance
|
|
||||||
|
|
||||||
Removed Sections: N/A (initial version)
|
|
||||||
|
|
||||||
Templates Status:
|
|
||||||
✅ plan-template.md - Reviewed, aligned with modular architecture and testing principles
|
|
||||||
✅ spec-template.md - Reviewed, aligned with user scenario focus and requirements structure
|
|
||||||
✅ tasks-template.md - Reviewed, aligned with test-driven and parallel development principles
|
|
||||||
✅ agent-file-template.md - Reviewed, no agent-specific conflicts
|
|
||||||
✅ checklist-template.md - Reviewed, compatible with quality standards
|
|
||||||
|
|
||||||
Follow-up TODOs:
|
|
||||||
- None at this time
|
|
||||||
==================
|
|
||||||
-->
|
|
||||||
|
|
||||||
## Core Principles
|
|
||||||
|
|
||||||
### I. Modular Architecture
|
|
||||||
|
|
||||||
Nextflow MUST maintain a clear separation between core functionality and extensions through its modular architecture:
|
|
||||||
|
|
||||||
- **Core modules** (`modules/`) contain essential functionality: core workflow runtime & plugin system (nextflow), shared utilities (nf-commons), language parsing (nf-lang), HTTP filesystem support (nf-httpfs), and lineage tracking (nf-lineage)
|
|
||||||
- **Core plugins** (`plugins/`) provides cloud provider integrations (AWS, Azure, GCP), execution platforms (Kubernetes), and specialized services (Seqera Platform, Wave container management)
|
|
||||||
- New features MUST be evaluated for placement: core features belong in `modules/`, specialized/cloud-specific features belong in `plugins/`
|
|
||||||
- Each module and plugin MUST be independently buildable and testable
|
|
||||||
- Plugin dependencies MUST be explicitly declared in `build.gradle` with semantic versioning
|
|
||||||
|
|
||||||
**Rationale**: This architecture enables independent development of cloud provider features without core engine changes, supports third-party plugin development, and maintains a clean separation of concerns across a large multi-module codebase.
|
|
||||||
|
|
||||||
### II. Test-Driven Quality Assurance (NON-NEGOTIABLE)
|
|
||||||
|
|
||||||
Testing MUST be comprehensive and multi-layered before any code is merged:
|
|
||||||
|
|
||||||
- **Unit tests** MUST use Spock Framework for all Groovy code, be independently executable, and achieve meaningful coverage (measured via JaCoCo)
|
|
||||||
- **Integration tests** (`tests/` directory) MUST validate end-to-end workflows using actual `.nf` scripts with expected outputs
|
|
||||||
- **Smoke tests** (`make smoke` or `NXF_SMOKE=1`) MUST be available to skip long-running and cloud-dependent tests during rapid development
|
|
||||||
- **Cloud validation tests** (`validation/` directory) MUST verify cloud provider integrations end-to-end before release
|
|
||||||
- **Documentation tests** (`docs/snippets/`) MUST ensure all documentation examples remain functional
|
|
||||||
- All tests MUST pass before commits, and `make test` MUST be run locally before pushing
|
|
||||||
|
|
||||||
**Rationale**: Scientific workflows demand reliability and reproducibility. Multi-layered testing catches issues at appropriate levels: unit tests for logic, integration tests for workflow correctness, and validation tests for cloud provider compatibility.
|
|
||||||
|
|
||||||
### III. Dataflow Programming Model
|
|
||||||
|
|
||||||
Nextflow's core abstraction MUST adhere to following principles and guidelines:
|
|
||||||
|
|
||||||
- The dataflow programming model is the fundamental abstraction in the Nextflow programming model.
|
|
||||||
- Workflows are defined as dataflow graphs where data flows between processes.
|
|
||||||
- Processes MUST be stateless, side-effect-free transformations that communicate via channels.
|
|
||||||
- Workflows should be defined in a platform agnostic manner to enable portability across clusters and clouds.
|
|
||||||
- The DSL MUST prioritize expressiveness for concurrent and parallel pipeline definition.
|
|
||||||
- Changes to the language parser (ANTLR grammars in `nf-lang`) MUST preserve backward compatibility with existing pipelines unless explicitly versioned (DSL1 vs DSL2).
|
|
||||||
- Concurrency primitives (GPars actors/dataflow) MUST be used correctly to maintain the dataflow semantics.
|
|
||||||
|
|
||||||
**Rationale**: The dataflow model is Nextflow's fundamental abstraction, enabling automatic parallelization and distribution. Preserving this model ensures existing scientific pipelines continue to work and users can reason about workflow behavior.
|
|
||||||
|
|
||||||
### IV. Apache 2.0 License Compliance
|
|
||||||
|
|
||||||
All source code MUST include Apache 2.0 license headers:
|
|
||||||
|
|
||||||
- Every source file MUST begin with the Apache 2.0 license header
|
|
||||||
- All contributions MUST comply with Apache 2.0 terms
|
|
||||||
- Third-party dependencies MUST use compatible licenses
|
|
||||||
- License compliance MUST be verified during code review
|
|
||||||
|
|
||||||
**Rationale**: Legal clarity protects both contributors and users. Consistent licensing enables academic and commercial use, which is critical for scientific software adoption.
|
|
||||||
|
|
||||||
### V. Developer Certificate of Origin (DCO) Sign-off
|
|
||||||
|
|
||||||
All commits MUST be signed with DCO certification:
|
|
||||||
|
|
||||||
- Contributors MUST certify they have the right to submit the code by using `git commit -s` or `git commit --signoff`
|
|
||||||
- Every commit message MUST include a `Signed-off-by` line
|
|
||||||
- The DCO bot MUST verify sign-off before any PR can be merged
|
|
||||||
- Contributors MUST NOT bypass the DCO requirement
|
|
||||||
|
|
||||||
**Rationale**: DCO provides legal protection and clear chain of custody for contributions, which is essential for open-source projects with diverse contributors.
|
|
||||||
|
|
||||||
### VI. Semantic Versioning and Release Discipline
|
|
||||||
|
|
||||||
Version management MUST follow strict semantic versioning with calendar-based releases:
|
|
||||||
|
|
||||||
- **Project versions** use calendar-based scheme: `YY.MM.PATCH` where April (`.04.`) and October (`.10.`) are stable releases, all other months use `-edge` suffix (e.g., `25.09.0-edge`)
|
|
||||||
- **Plugin versions** MUST use semantic versioning (`MAJOR.MINOR.PATCH`)
|
|
||||||
- Version changes MUST be documented in `changelog.txt` files (both project root and per-plugin)
|
|
||||||
- Breaking changes MUST increment MAJOR version for plugins and be clearly documented
|
|
||||||
- Release process MUST follow the documented procedure in `CLAUDE.md` including: updating changelogs, version files, running `make releaseInfo`, using `[release]` tag in commit message
|
|
||||||
|
|
||||||
**Rationale**: Predictable versioning enables users to understand compatibility and stability expectations. Calendar-based versioning for the main project makes release timing transparent, while semantic versioning for plugins enables clear communication of breaking changes.
|
|
||||||
|
|
||||||
### VII. Groovy Idioms and Code Standards
|
|
||||||
|
|
||||||
Code MUST follow Groovy best practices and Nextflow conventions:
|
|
||||||
|
|
||||||
- Use Groovy idioms (closures, operator overloading, DSL builders) appropriately
|
|
||||||
- Follow existing code patterns and conventions from similar modules
|
|
||||||
- Leverage Groovy's dynamic capabilities judiciously without sacrificing type safety where beneficial
|
|
||||||
- Use Groovy's `@CompileStatic` where performance is critical or type safety is desired
|
|
||||||
- AST transformations (in `modules/nextflow`) MUST be well-documented due to their compile-time magic
|
|
||||||
- Code MUST be formatted consistently (consider CodeNarc configuration in `gradle/codenarc.groovy`)
|
|
||||||
|
|
||||||
**Rationale**: Groovy enables powerful DSL capabilities that make Nextflow's language expressive, but requires discipline to maintain readability and debuggability. Consistency across the large codebase improves maintainability.
|
|
||||||
|
|
||||||
## Development Workflow
|
|
||||||
|
|
||||||
### Build and Development Process
|
|
||||||
|
|
||||||
- **Build tool**: Gradle with wrapper (`./gradlew`) is the authoritative build system
|
|
||||||
- **Quick commands**: Makefile provides convenience targets (`make compile`, `make test`, `make assemble`, `make check`, `make clean`)
|
|
||||||
- **Development testing**: Use `./launch.sh run script.nf` for testing changes against real workflows without full installation
|
|
||||||
- **Local installation**: `make install` publishes to Maven local for integration testing
|
|
||||||
- **Dependency management**: All dependencies MUST be declared in `build.gradle` with explicit versions; use `make deps` to analyze dependency trees
|
|
||||||
|
|
||||||
### Git Workflow
|
|
||||||
|
|
||||||
- **Branch management**: Work on feature branches, never commit directly to `master`
|
|
||||||
- **Commit sign-off**: Always use `git commit -s` to add DCO sign-off
|
|
||||||
- **CI control tags**: Use special commit message tags to control CI behavior:
|
|
||||||
- `[ci skip]` - Skip CI tests entirely
|
|
||||||
- `[ci fast]` - Run only unit tests, skip integration tests
|
|
||||||
- `[e2e stage]` - Run end-to-end tests against Seqera platform staging environment
|
|
||||||
- `[e2e prod]` - Run end-to-end tests against production platform
|
|
||||||
- `[release]` - Trigger release automation
|
|
||||||
- **Pull requests**: Must pass all CI checks, require code review, and have DCO verification
|
|
||||||
|
|
||||||
### Architecture Decision Records (ADRs)
|
|
||||||
|
|
||||||
- Significant structural and technical decisions MUST be documented as ADRs in the `adr/` directory
|
|
||||||
- ADRs MUST follow the template format: date prefix + descriptive name (e.g., `20251114-module-system.md`)
|
|
||||||
- ADRs provide historical context for why architectural decisions were made
|
|
||||||
- When changing fundamental architecture, review existing ADRs and create new ones documenting the rationale
|
|
||||||
|
|
||||||
## Quality Standards
|
|
||||||
|
|
||||||
### Code Review Requirements
|
|
||||||
|
|
||||||
- All changes MUST go through pull request review
|
|
||||||
- Reviewers MUST verify:
|
|
||||||
- Tests are included and passing
|
|
||||||
- Code follows Groovy idioms and project conventions
|
|
||||||
- License headers are present
|
|
||||||
- DCO sign-off is present
|
|
||||||
- Changes align with modular architecture principles
|
|
||||||
- Breaking changes are appropriately versioned and documented
|
|
||||||
|
|
||||||
### Testing Gates
|
|
||||||
|
|
||||||
- `make test` MUST pass before committing locally
|
|
||||||
- All CI tests MUST pass before merging
|
|
||||||
- Integration tests MUST be run for changes affecting workflow execution
|
|
||||||
- Cloud validation tests MUST be run before releases touching cloud provider plugins
|
|
||||||
- Smoke tests enable rapid iteration but MUST NOT replace full test execution
|
|
||||||
|
|
||||||
### Performance and Compatibility
|
|
||||||
|
|
||||||
- Target platform: Java 17 runtime compatibility (development uses Java 21 toolchain)
|
|
||||||
- Performance-critical paths SHOULD be profiled and optimized
|
|
||||||
- Memory usage SHOULD be monitored for large-scale workflows
|
|
||||||
- Backward compatibility MUST be maintained for existing DSL features unless a new DSL version is introduced
|
|
||||||
|
|
||||||
## Governance
|
|
||||||
|
|
||||||
### Amendment Process
|
|
||||||
|
|
||||||
This constitution supersedes all other development practices. Amendments require:
|
|
||||||
|
|
||||||
1. **Proposal**: Submit amendment proposal via GitHub issue or pull request
|
|
||||||
2. **Discussion**: Community discussion period (minimum 1 week for major changes)
|
|
||||||
3. **Approval**: Approval from core maintainers
|
|
||||||
4. **Documentation**: Update this constitution with version bump following semantic versioning:
|
|
||||||
- **MAJOR**: Backward incompatible governance changes, principle removal/redefinition
|
|
||||||
- **MINOR**: New principle added or materially expanded guidance
|
|
||||||
- **PATCH**: Clarifications, wording improvements, typo fixes
|
|
||||||
|
|
||||||
### Compliance and Review
|
|
||||||
|
|
||||||
- All pull requests and code reviews MUST verify compliance with these principles
|
|
||||||
- Deviations from principles MUST be explicitly justified in PR description
|
|
||||||
- Complexity additions MUST be justified against the "simplicity first" principle
|
|
||||||
- Constitution compliance is enforced through code review and CI automation where possible
|
|
||||||
|
|
||||||
### Ratification and Version History
|
|
||||||
|
|
||||||
**Version**: 1.0.0 | **Ratified**: 2025-11-17 | **Last Amended**: 2025-11-17
|
|
||||||
|
|
||||||
This constitution was derived from the Nextflow project's documented practices in `CLAUDE.md`, `CONTRIBUTING.md`, and the project's existing architectural patterns. It codifies the development principles that have made Nextflow a successful scientific workflow management system.
|
|
||||||
@@ -1,166 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
|
|
||||||
# Consolidated prerequisite checking script
|
|
||||||
#
|
|
||||||
# This script provides unified prerequisite checking for Spec-Driven Development workflow.
|
|
||||||
# It replaces the functionality previously spread across multiple scripts.
|
|
||||||
#
|
|
||||||
# Usage: ./check-prerequisites.sh [OPTIONS]
|
|
||||||
#
|
|
||||||
# OPTIONS:
|
|
||||||
# --json Output in JSON format
|
|
||||||
# --require-tasks Require tasks.md to exist (for implementation phase)
|
|
||||||
# --include-tasks Include tasks.md in AVAILABLE_DOCS list
|
|
||||||
# --paths-only Only output path variables (no validation)
|
|
||||||
# --help, -h Show help message
|
|
||||||
#
|
|
||||||
# OUTPUTS:
|
|
||||||
# JSON mode: {"FEATURE_DIR":"...", "AVAILABLE_DOCS":["..."]}
|
|
||||||
# Text mode: FEATURE_DIR:... \n AVAILABLE_DOCS: \n ✓/✗ file.md
|
|
||||||
# Paths only: REPO_ROOT: ... \n BRANCH: ... \n FEATURE_DIR: ... etc.
|
|
||||||
|
|
||||||
set -e
|
|
||||||
|
|
||||||
# Parse command line arguments
|
|
||||||
JSON_MODE=false
|
|
||||||
REQUIRE_TASKS=false
|
|
||||||
INCLUDE_TASKS=false
|
|
||||||
PATHS_ONLY=false
|
|
||||||
|
|
||||||
for arg in "$@"; do
|
|
||||||
case "$arg" in
|
|
||||||
--json)
|
|
||||||
JSON_MODE=true
|
|
||||||
;;
|
|
||||||
--require-tasks)
|
|
||||||
REQUIRE_TASKS=true
|
|
||||||
;;
|
|
||||||
--include-tasks)
|
|
||||||
INCLUDE_TASKS=true
|
|
||||||
;;
|
|
||||||
--paths-only)
|
|
||||||
PATHS_ONLY=true
|
|
||||||
;;
|
|
||||||
--help|-h)
|
|
||||||
cat << 'EOF'
|
|
||||||
Usage: check-prerequisites.sh [OPTIONS]
|
|
||||||
|
|
||||||
Consolidated prerequisite checking for Spec-Driven Development workflow.
|
|
||||||
|
|
||||||
OPTIONS:
|
|
||||||
--json Output in JSON format
|
|
||||||
--require-tasks Require tasks.md to exist (for implementation phase)
|
|
||||||
--include-tasks Include tasks.md in AVAILABLE_DOCS list
|
|
||||||
--paths-only Only output path variables (no prerequisite validation)
|
|
||||||
--help, -h Show this help message
|
|
||||||
|
|
||||||
EXAMPLES:
|
|
||||||
# Check task prerequisites (plan.md required)
|
|
||||||
./check-prerequisites.sh --json
|
|
||||||
|
|
||||||
# Check implementation prerequisites (plan.md + tasks.md required)
|
|
||||||
./check-prerequisites.sh --json --require-tasks --include-tasks
|
|
||||||
|
|
||||||
# Get feature paths only (no validation)
|
|
||||||
./check-prerequisites.sh --paths-only
|
|
||||||
|
|
||||||
EOF
|
|
||||||
exit 0
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
echo "ERROR: Unknown option '$arg'. Use --help for usage information." >&2
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
|
|
||||||
# Source common functions
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
source "$SCRIPT_DIR/common.sh"
|
|
||||||
|
|
||||||
# Get feature paths and validate branch
|
|
||||||
eval $(get_feature_paths)
|
|
||||||
check_feature_branch "$CURRENT_BRANCH" "$HAS_GIT" || exit 1
|
|
||||||
|
|
||||||
# If paths-only mode, output paths and exit (support JSON + paths-only combined)
|
|
||||||
if $PATHS_ONLY; then
|
|
||||||
if $JSON_MODE; then
|
|
||||||
# Minimal JSON paths payload (no validation performed)
|
|
||||||
printf '{"REPO_ROOT":"%s","BRANCH":"%s","FEATURE_DIR":"%s","FEATURE_SPEC":"%s","IMPL_PLAN":"%s","TASKS":"%s"}\n' \
|
|
||||||
"$REPO_ROOT" "$CURRENT_BRANCH" "$FEATURE_DIR" "$FEATURE_SPEC" "$IMPL_PLAN" "$TASKS"
|
|
||||||
else
|
|
||||||
echo "REPO_ROOT: $REPO_ROOT"
|
|
||||||
echo "BRANCH: $CURRENT_BRANCH"
|
|
||||||
echo "FEATURE_DIR: $FEATURE_DIR"
|
|
||||||
echo "FEATURE_SPEC: $FEATURE_SPEC"
|
|
||||||
echo "IMPL_PLAN: $IMPL_PLAN"
|
|
||||||
echo "TASKS: $TASKS"
|
|
||||||
fi
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Validate required directories and files
|
|
||||||
if [[ ! -d "$FEATURE_DIR" ]]; then
|
|
||||||
echo "ERROR: Feature directory not found: $FEATURE_DIR" >&2
|
|
||||||
echo "Run /speckit.specify first to create the feature structure." >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ ! -f "$IMPL_PLAN" ]]; then
|
|
||||||
echo "ERROR: plan.md not found in $FEATURE_DIR" >&2
|
|
||||||
echo "Run /speckit.plan first to create the implementation plan." >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check for tasks.md if required
|
|
||||||
if $REQUIRE_TASKS && [[ ! -f "$TASKS" ]]; then
|
|
||||||
echo "ERROR: tasks.md not found in $FEATURE_DIR" >&2
|
|
||||||
echo "Run /speckit.tasks first to create the task list." >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Build list of available documents
|
|
||||||
docs=()
|
|
||||||
|
|
||||||
# Always check these optional docs
|
|
||||||
[[ -f "$RESEARCH" ]] && docs+=("research.md")
|
|
||||||
[[ -f "$DATA_MODEL" ]] && docs+=("data-model.md")
|
|
||||||
|
|
||||||
# Check contracts directory (only if it exists and has files)
|
|
||||||
if [[ -d "$CONTRACTS_DIR" ]] && [[ -n "$(ls -A "$CONTRACTS_DIR" 2>/dev/null)" ]]; then
|
|
||||||
docs+=("contracts/")
|
|
||||||
fi
|
|
||||||
|
|
||||||
[[ -f "$QUICKSTART" ]] && docs+=("quickstart.md")
|
|
||||||
|
|
||||||
# Include tasks.md if requested and it exists
|
|
||||||
if $INCLUDE_TASKS && [[ -f "$TASKS" ]]; then
|
|
||||||
docs+=("tasks.md")
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Output results
|
|
||||||
if $JSON_MODE; then
|
|
||||||
# Build JSON array of documents
|
|
||||||
if [[ ${#docs[@]} -eq 0 ]]; then
|
|
||||||
json_docs="[]"
|
|
||||||
else
|
|
||||||
json_docs=$(printf '"%s",' "${docs[@]}")
|
|
||||||
json_docs="[${json_docs%,}]"
|
|
||||||
fi
|
|
||||||
|
|
||||||
printf '{"FEATURE_DIR":"%s","AVAILABLE_DOCS":%s}\n' "$FEATURE_DIR" "$json_docs"
|
|
||||||
else
|
|
||||||
# Text output
|
|
||||||
echo "FEATURE_DIR:$FEATURE_DIR"
|
|
||||||
echo "AVAILABLE_DOCS:"
|
|
||||||
|
|
||||||
# Show status of each potential document
|
|
||||||
check_file "$RESEARCH" "research.md"
|
|
||||||
check_file "$DATA_MODEL" "data-model.md"
|
|
||||||
check_dir "$CONTRACTS_DIR" "contracts/"
|
|
||||||
check_file "$QUICKSTART" "quickstart.md"
|
|
||||||
|
|
||||||
if $INCLUDE_TASKS; then
|
|
||||||
check_file "$TASKS" "tasks.md"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
@@ -1,157 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# Common functions and variables for all scripts
|
|
||||||
|
|
||||||
# Get repository root, with fallback for non-git repositories
|
|
||||||
get_repo_root() {
|
|
||||||
if git rev-parse --show-toplevel >/dev/null 2>&1; then
|
|
||||||
git rev-parse --show-toplevel
|
|
||||||
else
|
|
||||||
# Fall back to script location for non-git repos
|
|
||||||
local script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
(cd "$script_dir/../../.." && pwd)
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# Get current branch, with fallback for non-git repositories
|
|
||||||
get_current_branch() {
|
|
||||||
# First check if SPECIFY_FEATURE environment variable is set
|
|
||||||
if [[ -n "${SPECIFY_FEATURE:-}" ]]; then
|
|
||||||
echo "$SPECIFY_FEATURE"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Then check git if available
|
|
||||||
if git rev-parse --abbrev-ref HEAD >/dev/null 2>&1; then
|
|
||||||
git rev-parse --abbrev-ref HEAD
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
# For non-git repos, try to find the latest feature directory
|
|
||||||
local repo_root=$(get_repo_root)
|
|
||||||
local specs_dir="$repo_root/specs"
|
|
||||||
|
|
||||||
if [[ -d "$specs_dir" ]]; then
|
|
||||||
local latest_feature=""
|
|
||||||
local highest=0
|
|
||||||
|
|
||||||
for dir in "$specs_dir"/*; do
|
|
||||||
if [[ -d "$dir" ]]; then
|
|
||||||
local dirname=$(basename "$dir")
|
|
||||||
if [[ "$dirname" =~ ^([0-9]{6})- ]]; then
|
|
||||||
local number=${BASH_REMATCH[1]}
|
|
||||||
number=$((10#$number))
|
|
||||||
if [[ "$number" -gt "$highest" ]]; then
|
|
||||||
highest=$number
|
|
||||||
latest_feature=$dirname
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
if [[ -n "$latest_feature" ]]; then
|
|
||||||
echo "$latest_feature"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "main" # Final fallback
|
|
||||||
}
|
|
||||||
|
|
||||||
# Check if we have git available
|
|
||||||
has_git() {
|
|
||||||
git rev-parse --show-toplevel >/dev/null 2>&1
|
|
||||||
}
|
|
||||||
|
|
||||||
check_feature_branch() {
|
|
||||||
local branch="$1"
|
|
||||||
local has_git_repo="$2"
|
|
||||||
|
|
||||||
# For non-git repos, we can't enforce branch naming but still provide output
|
|
||||||
if [[ "$has_git_repo" != "true" ]]; then
|
|
||||||
echo "[specify] Warning: Git repository not detected; skipped branch validation" >&2
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Accept date-based format (YYMMDD-feature)
|
|
||||||
if [[ ! "$branch" =~ ^[0-9]{6}- ]]; then
|
|
||||||
echo "ERROR: Not on a feature branch. Current branch: $branch" >&2
|
|
||||||
echo "Feature branches should be named like: YYMMDD-feature-name (e.g., 251030-nextflow-modules)" >&2
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
get_feature_dir() { echo "$1/specs/$2"; }
|
|
||||||
|
|
||||||
# Find feature directory by numeric prefix instead of exact branch match
|
|
||||||
# This allows multiple branches to work on the same spec (e.g., 251030-fix-bug, 251030-add-feature)
|
|
||||||
find_feature_dir_by_prefix() {
|
|
||||||
local repo_root="$1"
|
|
||||||
local branch_name="$2"
|
|
||||||
local specs_dir="$repo_root/specs"
|
|
||||||
|
|
||||||
# Extract numeric prefix from branch (e.g., "251030" from "251030-whatever")
|
|
||||||
if [[ ! "$branch_name" =~ ^([0-9]{6})- ]]; then
|
|
||||||
# If branch doesn't have numeric prefix, fall back to exact match
|
|
||||||
echo "$specs_dir/$branch_name"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
local prefix="${BASH_REMATCH[1]}"
|
|
||||||
|
|
||||||
# Search for directories in specs/ that start with this prefix
|
|
||||||
local matches=()
|
|
||||||
if [[ -d "$specs_dir" ]]; then
|
|
||||||
for dir in "$specs_dir"/"$prefix"-*; do
|
|
||||||
if [[ -d "$dir" ]]; then
|
|
||||||
matches+=("$(basename "$dir")")
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Handle results
|
|
||||||
if [[ ${#matches[@]} -eq 0 ]]; then
|
|
||||||
# No match found - return the branch name path (will fail later with clear error)
|
|
||||||
echo "$specs_dir/$branch_name"
|
|
||||||
elif [[ ${#matches[@]} -eq 1 ]]; then
|
|
||||||
# Exactly one match - perfect!
|
|
||||||
echo "$specs_dir/${matches[0]}"
|
|
||||||
else
|
|
||||||
# Multiple matches - this shouldn't happen with proper naming convention
|
|
||||||
echo "ERROR: Multiple spec directories found with prefix '$prefix': ${matches[*]}" >&2
|
|
||||||
echo "Please ensure only one spec directory exists per numeric prefix." >&2
|
|
||||||
echo "$specs_dir/$branch_name" # Return something to avoid breaking the script
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
get_feature_paths() {
|
|
||||||
local repo_root=$(get_repo_root)
|
|
||||||
local current_branch=$(get_current_branch)
|
|
||||||
local has_git_repo="false"
|
|
||||||
|
|
||||||
if has_git; then
|
|
||||||
has_git_repo="true"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Use prefix-based lookup to support multiple branches per spec
|
|
||||||
local feature_dir=$(find_feature_dir_by_prefix "$repo_root" "$current_branch")
|
|
||||||
|
|
||||||
cat <<EOF
|
|
||||||
REPO_ROOT='$repo_root'
|
|
||||||
CURRENT_BRANCH='$current_branch'
|
|
||||||
HAS_GIT='$has_git_repo'
|
|
||||||
FEATURE_DIR='$feature_dir'
|
|
||||||
FEATURE_SPEC='$feature_dir/spec.md'
|
|
||||||
IMPL_PLAN='$feature_dir/plan.md'
|
|
||||||
TASKS='$feature_dir/tasks.md'
|
|
||||||
RESEARCH='$feature_dir/research.md'
|
|
||||||
DATA_MODEL='$feature_dir/data-model.md'
|
|
||||||
QUICKSTART='$feature_dir/quickstart.md'
|
|
||||||
CONTRACTS_DIR='$feature_dir/contracts'
|
|
||||||
EOF
|
|
||||||
}
|
|
||||||
|
|
||||||
check_file() { [[ -f "$1" ]] && echo " ✓ $2" || echo " ✗ $2"; }
|
|
||||||
check_dir() { [[ -d "$1" && -n $(ls -A "$1" 2>/dev/null) ]] && echo " ✓ $2" || echo " ✗ $2"; }
|
|
||||||
|
|
||||||
@@ -1,200 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
|
|
||||||
set -e
|
|
||||||
|
|
||||||
JSON_MODE=false
|
|
||||||
SHORT_NAME=""
|
|
||||||
ARGS=()
|
|
||||||
i=1
|
|
||||||
while [ $i -le $# ]; do
|
|
||||||
arg="${!i}"
|
|
||||||
case "$arg" in
|
|
||||||
--json)
|
|
||||||
JSON_MODE=true
|
|
||||||
;;
|
|
||||||
--short-name)
|
|
||||||
if [ $((i + 1)) -gt $# ]; then
|
|
||||||
echo 'Error: --short-name requires a value' >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
i=$((i + 1))
|
|
||||||
next_arg="${!i}"
|
|
||||||
# Check if the next argument is another option (starts with --)
|
|
||||||
if [[ "$next_arg" == --* ]]; then
|
|
||||||
echo 'Error: --short-name requires a value' >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
SHORT_NAME="$next_arg"
|
|
||||||
;;
|
|
||||||
--help|-h)
|
|
||||||
echo "Usage: $0 [--json] [--short-name <name>] <feature_description>"
|
|
||||||
echo ""
|
|
||||||
echo "Options:"
|
|
||||||
echo " --json Output in JSON format"
|
|
||||||
echo " --short-name <name> Provide a custom short name (2-4 words) for the branch"
|
|
||||||
echo " --help, -h Show this help message"
|
|
||||||
echo ""
|
|
||||||
echo "Branch naming: Uses YYMMDD-<feature-name> format (e.g., 251023-user-auth)"
|
|
||||||
echo ""
|
|
||||||
echo "Examples:"
|
|
||||||
echo " $0 'Add user authentication system' --short-name 'user-auth'"
|
|
||||||
echo " # Creates: 251023-user-auth (if run on Oct 23, 2025)"
|
|
||||||
echo ""
|
|
||||||
echo " $0 'Implement OAuth2 integration for API'"
|
|
||||||
echo " # Creates: 251023-oauth2-integration-api"
|
|
||||||
exit 0
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
ARGS+=("$arg")
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
i=$((i + 1))
|
|
||||||
done
|
|
||||||
|
|
||||||
FEATURE_DESCRIPTION="${ARGS[*]}"
|
|
||||||
if [ -z "$FEATURE_DESCRIPTION" ]; then
|
|
||||||
echo "Usage: $0 [--json] [--short-name <name>] <feature_description>" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Function to find the repository root by searching for existing project markers
|
|
||||||
find_repo_root() {
|
|
||||||
local dir="$1"
|
|
||||||
while [ "$dir" != "/" ]; do
|
|
||||||
if [ -d "$dir/.git" ] || [ -d "$dir/.specify" ]; then
|
|
||||||
echo "$dir"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
dir="$(dirname "$dir")"
|
|
||||||
done
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
# Resolve repository root. Prefer git information when available, but fall back
|
|
||||||
# to searching for repository markers so the workflow still functions in repositories that
|
|
||||||
# were initialised with --no-git.
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
|
|
||||||
if git rev-parse --show-toplevel >/dev/null 2>&1; then
|
|
||||||
REPO_ROOT=$(git rev-parse --show-toplevel)
|
|
||||||
HAS_GIT=true
|
|
||||||
else
|
|
||||||
REPO_ROOT="$(find_repo_root "$SCRIPT_DIR")"
|
|
||||||
if [ -z "$REPO_ROOT" ]; then
|
|
||||||
echo "Error: Could not determine repository root. Please run this script from within the repository." >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
HAS_GIT=false
|
|
||||||
fi
|
|
||||||
|
|
||||||
cd "$REPO_ROOT"
|
|
||||||
|
|
||||||
SPECS_DIR="$REPO_ROOT/specs"
|
|
||||||
mkdir -p "$SPECS_DIR"
|
|
||||||
|
|
||||||
# Use YYMMDD format for feature numbering (date-based prefix)
|
|
||||||
FEATURE_NUM=$(date +%y%m%d)
|
|
||||||
|
|
||||||
# Function to generate branch name with stop word filtering and length filtering
|
|
||||||
generate_branch_name() {
|
|
||||||
local description="$1"
|
|
||||||
|
|
||||||
# Common stop words to filter out
|
|
||||||
local stop_words="^(i|a|an|the|to|for|of|in|on|at|by|with|from|is|are|was|were|be|been|being|have|has|had|do|does|did|will|would|should|could|can|may|might|must|shall|this|that|these|those|my|your|our|their|want|need|add|get|set)$"
|
|
||||||
|
|
||||||
# Convert to lowercase and split into words
|
|
||||||
local clean_name=$(echo "$description" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/ /g')
|
|
||||||
|
|
||||||
# Filter words: remove stop words and words shorter than 3 chars (unless they're uppercase acronyms in original)
|
|
||||||
local meaningful_words=()
|
|
||||||
for word in $clean_name; do
|
|
||||||
# Skip empty words
|
|
||||||
[ -z "$word" ] && continue
|
|
||||||
|
|
||||||
# Keep words that are NOT stop words AND (length >= 3 OR are potential acronyms)
|
|
||||||
if ! echo "$word" | grep -qiE "$stop_words"; then
|
|
||||||
if [ ${#word} -ge 3 ]; then
|
|
||||||
meaningful_words+=("$word")
|
|
||||||
elif echo "$description" | grep -q "\b${word^^}\b"; then
|
|
||||||
# Keep short words if they appear as uppercase in original (likely acronyms)
|
|
||||||
meaningful_words+=("$word")
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# If we have meaningful words, use first 3-4 of them
|
|
||||||
if [ ${#meaningful_words[@]} -gt 0 ]; then
|
|
||||||
local max_words=3
|
|
||||||
if [ ${#meaningful_words[@]} -eq 4 ]; then max_words=4; fi
|
|
||||||
|
|
||||||
local result=""
|
|
||||||
local count=0
|
|
||||||
for word in "${meaningful_words[@]}"; do
|
|
||||||
if [ $count -ge $max_words ]; then break; fi
|
|
||||||
if [ -n "$result" ]; then result="$result-"; fi
|
|
||||||
result="$result$word"
|
|
||||||
count=$((count + 1))
|
|
||||||
done
|
|
||||||
echo "$result"
|
|
||||||
else
|
|
||||||
# Fallback to original logic if no meaningful words found
|
|
||||||
echo "$description" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/-\+/-/g' | sed 's/^-//' | sed 's/-$//' | tr '-' '\n' | grep -v '^$' | head -3 | tr '\n' '-' | sed 's/-$//'
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# Generate branch name
|
|
||||||
if [ -n "$SHORT_NAME" ]; then
|
|
||||||
# Use provided short name, just clean it up
|
|
||||||
BRANCH_SUFFIX=$(echo "$SHORT_NAME" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/-\+/-/g' | sed 's/^-//' | sed 's/-$//')
|
|
||||||
else
|
|
||||||
# Generate from description with smart filtering
|
|
||||||
BRANCH_SUFFIX=$(generate_branch_name "$FEATURE_DESCRIPTION")
|
|
||||||
fi
|
|
||||||
|
|
||||||
BRANCH_NAME="${FEATURE_NUM}-${BRANCH_SUFFIX}"
|
|
||||||
|
|
||||||
# GitHub enforces a 244-byte limit on branch names
|
|
||||||
# Validate and truncate if necessary
|
|
||||||
MAX_BRANCH_LENGTH=244
|
|
||||||
if [ ${#BRANCH_NAME} -gt $MAX_BRANCH_LENGTH ]; then
|
|
||||||
# Calculate how much we need to trim from suffix
|
|
||||||
# Account for: feature number (3) + hyphen (1) = 4 chars
|
|
||||||
MAX_SUFFIX_LENGTH=$((MAX_BRANCH_LENGTH - 4))
|
|
||||||
|
|
||||||
# Truncate suffix at word boundary if possible
|
|
||||||
TRUNCATED_SUFFIX=$(echo "$BRANCH_SUFFIX" | cut -c1-$MAX_SUFFIX_LENGTH)
|
|
||||||
# Remove trailing hyphen if truncation created one
|
|
||||||
TRUNCATED_SUFFIX=$(echo "$TRUNCATED_SUFFIX" | sed 's/-$//')
|
|
||||||
|
|
||||||
ORIGINAL_BRANCH_NAME="$BRANCH_NAME"
|
|
||||||
BRANCH_NAME="${FEATURE_NUM}-${TRUNCATED_SUFFIX}"
|
|
||||||
|
|
||||||
>&2 echo "[specify] Warning: Branch name exceeded GitHub's 244-byte limit"
|
|
||||||
>&2 echo "[specify] Original: $ORIGINAL_BRANCH_NAME (${#ORIGINAL_BRANCH_NAME} bytes)"
|
|
||||||
>&2 echo "[specify] Truncated to: $BRANCH_NAME (${#BRANCH_NAME} bytes)"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ "$HAS_GIT" = true ]; then
|
|
||||||
git checkout -b "$BRANCH_NAME"
|
|
||||||
else
|
|
||||||
>&2 echo "[specify] Warning: Git repository not detected; skipped branch creation for $BRANCH_NAME"
|
|
||||||
fi
|
|
||||||
|
|
||||||
FEATURE_DIR="$SPECS_DIR/$BRANCH_NAME"
|
|
||||||
mkdir -p "$FEATURE_DIR"
|
|
||||||
|
|
||||||
TEMPLATE="$REPO_ROOT/.specify/templates/spec-template.md"
|
|
||||||
SPEC_FILE="$FEATURE_DIR/spec.md"
|
|
||||||
if [ -f "$TEMPLATE" ]; then cp "$TEMPLATE" "$SPEC_FILE"; else touch "$SPEC_FILE"; fi
|
|
||||||
|
|
||||||
# Set the SPECIFY_FEATURE environment variable for the current session
|
|
||||||
export SPECIFY_FEATURE="$BRANCH_NAME"
|
|
||||||
|
|
||||||
if $JSON_MODE; then
|
|
||||||
printf '{"BRANCH_NAME":"%s","SPEC_FILE":"%s","FEATURE_NUM":"%s"}\n' "$BRANCH_NAME" "$SPEC_FILE" "$FEATURE_NUM"
|
|
||||||
else
|
|
||||||
echo "BRANCH_NAME: $BRANCH_NAME"
|
|
||||||
echo "SPEC_FILE: $SPEC_FILE"
|
|
||||||
echo "FEATURE_NUM: $FEATURE_NUM"
|
|
||||||
echo "SPECIFY_FEATURE environment variable set to: $BRANCH_NAME"
|
|
||||||
fi
|
|
||||||
@@ -1,61 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
|
|
||||||
set -e
|
|
||||||
|
|
||||||
# Parse command line arguments
|
|
||||||
JSON_MODE=false
|
|
||||||
ARGS=()
|
|
||||||
|
|
||||||
for arg in "$@"; do
|
|
||||||
case "$arg" in
|
|
||||||
--json)
|
|
||||||
JSON_MODE=true
|
|
||||||
;;
|
|
||||||
--help|-h)
|
|
||||||
echo "Usage: $0 [--json]"
|
|
||||||
echo " --json Output results in JSON format"
|
|
||||||
echo " --help Show this help message"
|
|
||||||
exit 0
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
ARGS+=("$arg")
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
|
|
||||||
# Get script directory and load common functions
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
source "$SCRIPT_DIR/common.sh"
|
|
||||||
|
|
||||||
# Get all paths and variables from common functions
|
|
||||||
eval $(get_feature_paths)
|
|
||||||
|
|
||||||
# Check if we're on a proper feature branch (only for git repos)
|
|
||||||
check_feature_branch "$CURRENT_BRANCH" "$HAS_GIT" || exit 1
|
|
||||||
|
|
||||||
# Ensure the feature directory exists
|
|
||||||
mkdir -p "$FEATURE_DIR"
|
|
||||||
|
|
||||||
# Copy plan template if it exists
|
|
||||||
TEMPLATE="$REPO_ROOT/.specify/templates/plan-template.md"
|
|
||||||
if [[ -f "$TEMPLATE" ]]; then
|
|
||||||
cp "$TEMPLATE" "$IMPL_PLAN"
|
|
||||||
echo "Copied plan template to $IMPL_PLAN"
|
|
||||||
else
|
|
||||||
echo "Warning: Plan template not found at $TEMPLATE"
|
|
||||||
# Create a basic plan file if template doesn't exist
|
|
||||||
touch "$IMPL_PLAN"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Output results
|
|
||||||
if $JSON_MODE; then
|
|
||||||
printf '{"FEATURE_SPEC":"%s","IMPL_PLAN":"%s","SPECS_DIR":"%s","BRANCH":"%s","HAS_GIT":"%s"}\n' \
|
|
||||||
"$FEATURE_SPEC" "$IMPL_PLAN" "$FEATURE_DIR" "$CURRENT_BRANCH" "$HAS_GIT"
|
|
||||||
else
|
|
||||||
echo "FEATURE_SPEC: $FEATURE_SPEC"
|
|
||||||
echo "IMPL_PLAN: $IMPL_PLAN"
|
|
||||||
echo "SPECS_DIR: $FEATURE_DIR"
|
|
||||||
echo "BRANCH: $CURRENT_BRANCH"
|
|
||||||
echo "HAS_GIT: $HAS_GIT"
|
|
||||||
fi
|
|
||||||
|
|
||||||
@@ -1,772 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
|
|
||||||
# Update agent context files with information from plan.md
|
|
||||||
#
|
|
||||||
# This script maintains AI agent context files by parsing feature specifications
|
|
||||||
# and updating agent-specific configuration files with project information.
|
|
||||||
#
|
|
||||||
# MAIN FUNCTIONS:
|
|
||||||
# 1. Environment Validation
|
|
||||||
# - Verifies git repository structure and branch information
|
|
||||||
# - Checks for required plan.md files and templates
|
|
||||||
# - Validates file permissions and accessibility
|
|
||||||
#
|
|
||||||
# 2. Plan Data Extraction
|
|
||||||
# - Parses plan.md files to extract project metadata
|
|
||||||
# - Identifies language/version, frameworks, databases, and project types
|
|
||||||
# - Handles missing or incomplete specification data gracefully
|
|
||||||
#
|
|
||||||
# 3. Agent File Management
|
|
||||||
# - Creates new agent context files from templates when needed
|
|
||||||
# - Updates existing agent files with new project information
|
|
||||||
# - Preserves manual additions and custom configurations
|
|
||||||
# - Supports multiple AI agent formats and directory structures
|
|
||||||
#
|
|
||||||
# 4. Content Generation
|
|
||||||
# - Generates language-specific build/test commands
|
|
||||||
# - Creates appropriate project directory structures
|
|
||||||
# - Updates technology stacks and recent changes sections
|
|
||||||
# - Maintains consistent formatting and timestamps
|
|
||||||
#
|
|
||||||
# 5. Multi-Agent Support
|
|
||||||
# - Handles agent-specific file paths and naming conventions
|
|
||||||
# - Supports: Claude, Gemini, Copilot, Cursor, Qwen, opencode, Codex, Windsurf, Kilo Code, Auggie CLI, Roo Code, CodeBuddy CLI, Amp, or Amazon Q Developer CLI
|
|
||||||
# - Can update single agents or all existing agent files
|
|
||||||
# - Creates default Claude file if no agent files exist
|
|
||||||
#
|
|
||||||
# Usage: ./update-agent-context.sh [agent_type]
|
|
||||||
# Agent types: claude|gemini|copilot|cursor-agent|qwen|opencode|codex|windsurf|kilocode|auggie|q
|
|
||||||
# Leave empty to update all existing agent files
|
|
||||||
|
|
||||||
set -e
|
|
||||||
|
|
||||||
# Enable strict error handling
|
|
||||||
set -u
|
|
||||||
set -o pipefail
|
|
||||||
|
|
||||||
#==============================================================================
|
|
||||||
# Configuration and Global Variables
|
|
||||||
#==============================================================================
|
|
||||||
|
|
||||||
# Get script directory and load common functions
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
source "$SCRIPT_DIR/common.sh"
|
|
||||||
|
|
||||||
# Get all paths and variables from common functions
|
|
||||||
eval $(get_feature_paths)
|
|
||||||
|
|
||||||
NEW_PLAN="$IMPL_PLAN" # Alias for compatibility with existing code
|
|
||||||
AGENT_TYPE="${1:-}"
|
|
||||||
|
|
||||||
# Agent-specific file paths
|
|
||||||
CLAUDE_FILE="$REPO_ROOT/CLAUDE.md"
|
|
||||||
GEMINI_FILE="$REPO_ROOT/GEMINI.md"
|
|
||||||
COPILOT_FILE="$REPO_ROOT/.github/copilot-instructions.md"
|
|
||||||
CURSOR_FILE="$REPO_ROOT/.cursor/rules/specify-rules.mdc"
|
|
||||||
QWEN_FILE="$REPO_ROOT/QWEN.md"
|
|
||||||
AGENTS_FILE="$REPO_ROOT/AGENTS.md"
|
|
||||||
WINDSURF_FILE="$REPO_ROOT/.windsurf/rules/specify-rules.md"
|
|
||||||
KILOCODE_FILE="$REPO_ROOT/.kilocode/rules/specify-rules.md"
|
|
||||||
AUGGIE_FILE="$REPO_ROOT/.augment/rules/specify-rules.md"
|
|
||||||
ROO_FILE="$REPO_ROOT/.roo/rules/specify-rules.md"
|
|
||||||
CODEBUDDY_FILE="$REPO_ROOT/CODEBUDDY.md"
|
|
||||||
AMP_FILE="$REPO_ROOT/AGENTS.md"
|
|
||||||
Q_FILE="$REPO_ROOT/AGENTS.md"
|
|
||||||
|
|
||||||
# Template file
|
|
||||||
TEMPLATE_FILE="$REPO_ROOT/.specify/templates/agent-file-template.md"
|
|
||||||
|
|
||||||
# Global variables for parsed plan data
|
|
||||||
NEW_LANG=""
|
|
||||||
NEW_FRAMEWORK=""
|
|
||||||
NEW_DB=""
|
|
||||||
NEW_PROJECT_TYPE=""
|
|
||||||
|
|
||||||
#==============================================================================
|
|
||||||
# Utility Functions
|
|
||||||
#==============================================================================
|
|
||||||
|
|
||||||
log_info() {
|
|
||||||
echo "INFO: $1"
|
|
||||||
}
|
|
||||||
|
|
||||||
log_success() {
|
|
||||||
echo "✓ $1"
|
|
||||||
}
|
|
||||||
|
|
||||||
log_error() {
|
|
||||||
echo "ERROR: $1" >&2
|
|
||||||
}
|
|
||||||
|
|
||||||
log_warning() {
|
|
||||||
echo "WARNING: $1" >&2
|
|
||||||
}
|
|
||||||
|
|
||||||
# Cleanup function for temporary files
|
|
||||||
cleanup() {
|
|
||||||
local exit_code=$?
|
|
||||||
rm -f /tmp/agent_update_*_$$
|
|
||||||
rm -f /tmp/manual_additions_$$
|
|
||||||
exit $exit_code
|
|
||||||
}
|
|
||||||
|
|
||||||
# Set up cleanup trap
|
|
||||||
trap cleanup EXIT INT TERM
|
|
||||||
|
|
||||||
#==============================================================================
|
|
||||||
# Validation Functions
|
|
||||||
#==============================================================================
|
|
||||||
|
|
||||||
validate_environment() {
|
|
||||||
# Check if we have a current branch/feature (git or non-git)
|
|
||||||
if [[ -z "$CURRENT_BRANCH" ]]; then
|
|
||||||
log_error "Unable to determine current feature"
|
|
||||||
if [[ "$HAS_GIT" == "true" ]]; then
|
|
||||||
log_info "Make sure you're on a feature branch"
|
|
||||||
else
|
|
||||||
log_info "Set SPECIFY_FEATURE environment variable or create a feature first"
|
|
||||||
fi
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check if plan.md exists
|
|
||||||
if [[ ! -f "$NEW_PLAN" ]]; then
|
|
||||||
log_error "No plan.md found at $NEW_PLAN"
|
|
||||||
log_info "Make sure you're working on a feature with a corresponding spec directory"
|
|
||||||
if [[ "$HAS_GIT" != "true" ]]; then
|
|
||||||
log_info "Use: export SPECIFY_FEATURE=your-feature-name or create a new feature first"
|
|
||||||
fi
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check if template exists (needed for new files)
|
|
||||||
if [[ ! -f "$TEMPLATE_FILE" ]]; then
|
|
||||||
log_warning "Template file not found at $TEMPLATE_FILE"
|
|
||||||
log_warning "Creating new agent files will fail"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
#==============================================================================
|
|
||||||
# Plan Parsing Functions
|
|
||||||
#==============================================================================
|
|
||||||
|
|
||||||
extract_plan_field() {
|
|
||||||
local field_pattern="$1"
|
|
||||||
local plan_file="$2"
|
|
||||||
|
|
||||||
grep "^\*\*${field_pattern}\*\*: " "$plan_file" 2>/dev/null | \
|
|
||||||
head -1 | \
|
|
||||||
sed "s|^\*\*${field_pattern}\*\*: ||" | \
|
|
||||||
sed 's/^[ \t]*//;s/[ \t]*$//' | \
|
|
||||||
grep -v "NEEDS CLARIFICATION" | \
|
|
||||||
grep -v "^N/A$" || echo ""
|
|
||||||
}
|
|
||||||
|
|
||||||
parse_plan_data() {
|
|
||||||
local plan_file="$1"
|
|
||||||
|
|
||||||
if [[ ! -f "$plan_file" ]]; then
|
|
||||||
log_error "Plan file not found: $plan_file"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ ! -r "$plan_file" ]]; then
|
|
||||||
log_error "Plan file is not readable: $plan_file"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
log_info "Parsing plan data from $plan_file"
|
|
||||||
|
|
||||||
NEW_LANG=$(extract_plan_field "Language/Version" "$plan_file")
|
|
||||||
NEW_FRAMEWORK=$(extract_plan_field "Primary Dependencies" "$plan_file")
|
|
||||||
NEW_DB=$(extract_plan_field "Storage" "$plan_file")
|
|
||||||
NEW_PROJECT_TYPE=$(extract_plan_field "Project Type" "$plan_file")
|
|
||||||
|
|
||||||
# Log what we found
|
|
||||||
if [[ -n "$NEW_LANG" ]]; then
|
|
||||||
log_info "Found language: $NEW_LANG"
|
|
||||||
else
|
|
||||||
log_warning "No language information found in plan"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -n "$NEW_FRAMEWORK" ]]; then
|
|
||||||
log_info "Found framework: $NEW_FRAMEWORK"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]]; then
|
|
||||||
log_info "Found database: $NEW_DB"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -n "$NEW_PROJECT_TYPE" ]]; then
|
|
||||||
log_info "Found project type: $NEW_PROJECT_TYPE"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
format_technology_stack() {
|
|
||||||
local lang="$1"
|
|
||||||
local framework="$2"
|
|
||||||
local parts=()
|
|
||||||
|
|
||||||
# Add non-empty parts
|
|
||||||
[[ -n "$lang" && "$lang" != "NEEDS CLARIFICATION" ]] && parts+=("$lang")
|
|
||||||
[[ -n "$framework" && "$framework" != "NEEDS CLARIFICATION" && "$framework" != "N/A" ]] && parts+=("$framework")
|
|
||||||
|
|
||||||
# Join with proper formatting
|
|
||||||
if [[ ${#parts[@]} -eq 0 ]]; then
|
|
||||||
echo ""
|
|
||||||
elif [[ ${#parts[@]} -eq 1 ]]; then
|
|
||||||
echo "${parts[0]}"
|
|
||||||
else
|
|
||||||
# Join multiple parts with " + "
|
|
||||||
local result="${parts[0]}"
|
|
||||||
for ((i=1; i<${#parts[@]}; i++)); do
|
|
||||||
result="$result + ${parts[i]}"
|
|
||||||
done
|
|
||||||
echo "$result"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
#==============================================================================
|
|
||||||
# Template and Content Generation Functions
|
|
||||||
#==============================================================================
|
|
||||||
|
|
||||||
get_project_structure() {
|
|
||||||
local project_type="$1"
|
|
||||||
|
|
||||||
if [[ "$project_type" == *"web"* ]]; then
|
|
||||||
echo "backend/\\nfrontend/\\ntests/"
|
|
||||||
else
|
|
||||||
echo "src/\\ntests/"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
get_commands_for_language() {
|
|
||||||
local lang="$1"
|
|
||||||
|
|
||||||
case "$lang" in
|
|
||||||
*"Python"*)
|
|
||||||
echo "cd src && pytest && ruff check ."
|
|
||||||
;;
|
|
||||||
*"Rust"*)
|
|
||||||
echo "cargo test && cargo clippy"
|
|
||||||
;;
|
|
||||||
*"JavaScript"*|*"TypeScript"*)
|
|
||||||
echo "npm test \\&\\& npm run lint"
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
echo "# Add commands for $lang"
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
get_language_conventions() {
|
|
||||||
local lang="$1"
|
|
||||||
echo "$lang: Follow standard conventions"
|
|
||||||
}
|
|
||||||
|
|
||||||
create_new_agent_file() {
|
|
||||||
local target_file="$1"
|
|
||||||
local temp_file="$2"
|
|
||||||
local project_name="$3"
|
|
||||||
local current_date="$4"
|
|
||||||
|
|
||||||
if [[ ! -f "$TEMPLATE_FILE" ]]; then
|
|
||||||
log_error "Template not found at $TEMPLATE_FILE"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ ! -r "$TEMPLATE_FILE" ]]; then
|
|
||||||
log_error "Template file is not readable: $TEMPLATE_FILE"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
log_info "Creating new agent context file from template..."
|
|
||||||
|
|
||||||
if ! cp "$TEMPLATE_FILE" "$temp_file"; then
|
|
||||||
log_error "Failed to copy template file"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Replace template placeholders
|
|
||||||
local project_structure
|
|
||||||
project_structure=$(get_project_structure "$NEW_PROJECT_TYPE")
|
|
||||||
|
|
||||||
local commands
|
|
||||||
commands=$(get_commands_for_language "$NEW_LANG")
|
|
||||||
|
|
||||||
local language_conventions
|
|
||||||
language_conventions=$(get_language_conventions "$NEW_LANG")
|
|
||||||
|
|
||||||
# Perform substitutions with error checking using safer approach
|
|
||||||
# Escape special characters for sed by using a different delimiter or escaping
|
|
||||||
local escaped_lang=$(printf '%s\n' "$NEW_LANG" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
|
||||||
local escaped_framework=$(printf '%s\n' "$NEW_FRAMEWORK" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
|
||||||
local escaped_branch=$(printf '%s\n' "$CURRENT_BRANCH" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
|
||||||
|
|
||||||
# Build technology stack and recent change strings conditionally
|
|
||||||
local tech_stack
|
|
||||||
if [[ -n "$escaped_lang" && -n "$escaped_framework" ]]; then
|
|
||||||
tech_stack="- $escaped_lang + $escaped_framework ($escaped_branch)"
|
|
||||||
elif [[ -n "$escaped_lang" ]]; then
|
|
||||||
tech_stack="- $escaped_lang ($escaped_branch)"
|
|
||||||
elif [[ -n "$escaped_framework" ]]; then
|
|
||||||
tech_stack="- $escaped_framework ($escaped_branch)"
|
|
||||||
else
|
|
||||||
tech_stack="- ($escaped_branch)"
|
|
||||||
fi
|
|
||||||
|
|
||||||
local recent_change
|
|
||||||
if [[ -n "$escaped_lang" && -n "$escaped_framework" ]]; then
|
|
||||||
recent_change="- $escaped_branch: Added $escaped_lang + $escaped_framework"
|
|
||||||
elif [[ -n "$escaped_lang" ]]; then
|
|
||||||
recent_change="- $escaped_branch: Added $escaped_lang"
|
|
||||||
elif [[ -n "$escaped_framework" ]]; then
|
|
||||||
recent_change="- $escaped_branch: Added $escaped_framework"
|
|
||||||
else
|
|
||||||
recent_change="- $escaped_branch: Added"
|
|
||||||
fi
|
|
||||||
|
|
||||||
local substitutions=(
|
|
||||||
"s|\[PROJECT NAME\]|$project_name|"
|
|
||||||
"s|\[DATE\]|$current_date|"
|
|
||||||
"s|\[EXTRACTED FROM ALL PLAN.MD FILES\]|$tech_stack|"
|
|
||||||
"s|\[ACTUAL STRUCTURE FROM PLANS\]|$project_structure|g"
|
|
||||||
"s|\[ONLY COMMANDS FOR ACTIVE TECHNOLOGIES\]|$commands|"
|
|
||||||
"s|\[LANGUAGE-SPECIFIC, ONLY FOR LANGUAGES IN USE\]|$language_conventions|"
|
|
||||||
"s|\[LAST 3 FEATURES AND WHAT THEY ADDED\]|$recent_change|"
|
|
||||||
)
|
|
||||||
|
|
||||||
for substitution in "${substitutions[@]}"; do
|
|
||||||
if ! sed -i.bak -e "$substitution" "$temp_file"; then
|
|
||||||
log_error "Failed to perform substitution: $substitution"
|
|
||||||
rm -f "$temp_file" "$temp_file.bak"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
# Convert \n sequences to actual newlines
|
|
||||||
newline=$(printf '\n')
|
|
||||||
sed -i.bak2 "s/\\\\n/${newline}/g" "$temp_file"
|
|
||||||
|
|
||||||
# Clean up backup files
|
|
||||||
rm -f "$temp_file.bak" "$temp_file.bak2"
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
update_existing_agent_file() {
|
|
||||||
local target_file="$1"
|
|
||||||
local current_date="$2"
|
|
||||||
|
|
||||||
log_info "Updating existing agent context file..."
|
|
||||||
|
|
||||||
# Use a single temporary file for atomic update
|
|
||||||
local temp_file
|
|
||||||
temp_file=$(mktemp) || {
|
|
||||||
log_error "Failed to create temporary file"
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
# Process the file in one pass
|
|
||||||
local tech_stack=$(format_technology_stack "$NEW_LANG" "$NEW_FRAMEWORK")
|
|
||||||
local new_tech_entries=()
|
|
||||||
local new_change_entry=""
|
|
||||||
|
|
||||||
# Prepare new technology entries
|
|
||||||
if [[ -n "$tech_stack" ]] && ! grep -q "$tech_stack" "$target_file"; then
|
|
||||||
new_tech_entries+=("- $tech_stack ($CURRENT_BRANCH)")
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]] && [[ "$NEW_DB" != "NEEDS CLARIFICATION" ]] && ! grep -q "$NEW_DB" "$target_file"; then
|
|
||||||
new_tech_entries+=("- $NEW_DB ($CURRENT_BRANCH)")
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Prepare new change entry
|
|
||||||
if [[ -n "$tech_stack" ]]; then
|
|
||||||
new_change_entry="- $CURRENT_BRANCH: Added $tech_stack"
|
|
||||||
elif [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]] && [[ "$NEW_DB" != "NEEDS CLARIFICATION" ]]; then
|
|
||||||
new_change_entry="- $CURRENT_BRANCH: Added $NEW_DB"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check if sections exist in the file
|
|
||||||
local has_active_technologies=0
|
|
||||||
local has_recent_changes=0
|
|
||||||
|
|
||||||
if grep -q "^## Active Technologies" "$target_file" 2>/dev/null; then
|
|
||||||
has_active_technologies=1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if grep -q "^## Recent Changes" "$target_file" 2>/dev/null; then
|
|
||||||
has_recent_changes=1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Process file line by line
|
|
||||||
local in_tech_section=false
|
|
||||||
local in_changes_section=false
|
|
||||||
local tech_entries_added=false
|
|
||||||
local changes_entries_added=false
|
|
||||||
local existing_changes_count=0
|
|
||||||
local file_ended=false
|
|
||||||
|
|
||||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
|
||||||
# Handle Active Technologies section
|
|
||||||
if [[ "$line" == "## Active Technologies" ]]; then
|
|
||||||
echo "$line" >> "$temp_file"
|
|
||||||
in_tech_section=true
|
|
||||||
continue
|
|
||||||
elif [[ $in_tech_section == true ]] && [[ "$line" =~ ^##[[:space:]] ]]; then
|
|
||||||
# Add new tech entries before closing the section
|
|
||||||
if [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
|
||||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
|
||||||
tech_entries_added=true
|
|
||||||
fi
|
|
||||||
echo "$line" >> "$temp_file"
|
|
||||||
in_tech_section=false
|
|
||||||
continue
|
|
||||||
elif [[ $in_tech_section == true ]] && [[ -z "$line" ]]; then
|
|
||||||
# Add new tech entries before empty line in tech section
|
|
||||||
if [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
|
||||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
|
||||||
tech_entries_added=true
|
|
||||||
fi
|
|
||||||
echo "$line" >> "$temp_file"
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Handle Recent Changes section
|
|
||||||
if [[ "$line" == "## Recent Changes" ]]; then
|
|
||||||
echo "$line" >> "$temp_file"
|
|
||||||
# Add new change entry right after the heading
|
|
||||||
if [[ -n "$new_change_entry" ]]; then
|
|
||||||
echo "$new_change_entry" >> "$temp_file"
|
|
||||||
fi
|
|
||||||
in_changes_section=true
|
|
||||||
changes_entries_added=true
|
|
||||||
continue
|
|
||||||
elif [[ $in_changes_section == true ]] && [[ "$line" =~ ^##[[:space:]] ]]; then
|
|
||||||
echo "$line" >> "$temp_file"
|
|
||||||
in_changes_section=false
|
|
||||||
continue
|
|
||||||
elif [[ $in_changes_section == true ]] && [[ "$line" == "- "* ]]; then
|
|
||||||
# Keep only first 2 existing changes
|
|
||||||
if [[ $existing_changes_count -lt 2 ]]; then
|
|
||||||
echo "$line" >> "$temp_file"
|
|
||||||
((existing_changes_count++))
|
|
||||||
fi
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Update timestamp
|
|
||||||
if [[ "$line" =~ \*\*Last\ updated\*\*:.*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] ]]; then
|
|
||||||
echo "$line" | sed "s/[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]/$current_date/" >> "$temp_file"
|
|
||||||
else
|
|
||||||
echo "$line" >> "$temp_file"
|
|
||||||
fi
|
|
||||||
done < "$target_file"
|
|
||||||
|
|
||||||
# Post-loop check: if we're still in the Active Technologies section and haven't added new entries
|
|
||||||
if [[ $in_tech_section == true ]] && [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
|
||||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
|
||||||
tech_entries_added=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
# If sections don't exist, add them at the end of the file
|
|
||||||
if [[ $has_active_technologies -eq 0 ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
|
||||||
echo "" >> "$temp_file"
|
|
||||||
echo "## Active Technologies" >> "$temp_file"
|
|
||||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
|
||||||
tech_entries_added=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ $has_recent_changes -eq 0 ]] && [[ -n "$new_change_entry" ]]; then
|
|
||||||
echo "" >> "$temp_file"
|
|
||||||
echo "## Recent Changes" >> "$temp_file"
|
|
||||||
echo "$new_change_entry" >> "$temp_file"
|
|
||||||
changes_entries_added=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Move temp file to target atomically
|
|
||||||
if ! mv "$temp_file" "$target_file"; then
|
|
||||||
log_error "Failed to update target file"
|
|
||||||
rm -f "$temp_file"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
#==============================================================================
|
|
||||||
# Main Agent File Update Function
|
|
||||||
#==============================================================================
|
|
||||||
|
|
||||||
update_agent_file() {
|
|
||||||
local target_file="$1"
|
|
||||||
local agent_name="$2"
|
|
||||||
|
|
||||||
if [[ -z "$target_file" ]] || [[ -z "$agent_name" ]]; then
|
|
||||||
log_error "update_agent_file requires target_file and agent_name parameters"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
log_info "Updating $agent_name context file: $target_file"
|
|
||||||
|
|
||||||
local project_name
|
|
||||||
project_name=$(basename "$REPO_ROOT")
|
|
||||||
local current_date
|
|
||||||
current_date=$(date +%Y-%m-%d)
|
|
||||||
|
|
||||||
# Create directory if it doesn't exist
|
|
||||||
local target_dir
|
|
||||||
target_dir=$(dirname "$target_file")
|
|
||||||
if [[ ! -d "$target_dir" ]]; then
|
|
||||||
if ! mkdir -p "$target_dir"; then
|
|
||||||
log_error "Failed to create directory: $target_dir"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ ! -f "$target_file" ]]; then
|
|
||||||
# Create new file from template
|
|
||||||
local temp_file
|
|
||||||
temp_file=$(mktemp) || {
|
|
||||||
log_error "Failed to create temporary file"
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
if create_new_agent_file "$target_file" "$temp_file" "$project_name" "$current_date"; then
|
|
||||||
if mv "$temp_file" "$target_file"; then
|
|
||||||
log_success "Created new $agent_name context file"
|
|
||||||
else
|
|
||||||
log_error "Failed to move temporary file to $target_file"
|
|
||||||
rm -f "$temp_file"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
log_error "Failed to create new agent file"
|
|
||||||
rm -f "$temp_file"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
# Update existing file
|
|
||||||
if [[ ! -r "$target_file" ]]; then
|
|
||||||
log_error "Cannot read existing file: $target_file"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ ! -w "$target_file" ]]; then
|
|
||||||
log_error "Cannot write to existing file: $target_file"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if update_existing_agent_file "$target_file" "$current_date"; then
|
|
||||||
log_success "Updated existing $agent_name context file"
|
|
||||||
else
|
|
||||||
log_error "Failed to update existing agent file"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
#==============================================================================
|
|
||||||
# Agent Selection and Processing
|
|
||||||
#==============================================================================
|
|
||||||
|
|
||||||
update_specific_agent() {
|
|
||||||
local agent_type="$1"
|
|
||||||
|
|
||||||
case "$agent_type" in
|
|
||||||
claude)
|
|
||||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
|
||||||
;;
|
|
||||||
gemini)
|
|
||||||
update_agent_file "$GEMINI_FILE" "Gemini CLI"
|
|
||||||
;;
|
|
||||||
copilot)
|
|
||||||
update_agent_file "$COPILOT_FILE" "GitHub Copilot"
|
|
||||||
;;
|
|
||||||
cursor-agent)
|
|
||||||
update_agent_file "$CURSOR_FILE" "Cursor IDE"
|
|
||||||
;;
|
|
||||||
qwen)
|
|
||||||
update_agent_file "$QWEN_FILE" "Qwen Code"
|
|
||||||
;;
|
|
||||||
opencode)
|
|
||||||
update_agent_file "$AGENTS_FILE" "opencode"
|
|
||||||
;;
|
|
||||||
codex)
|
|
||||||
update_agent_file "$AGENTS_FILE" "Codex CLI"
|
|
||||||
;;
|
|
||||||
windsurf)
|
|
||||||
update_agent_file "$WINDSURF_FILE" "Windsurf"
|
|
||||||
;;
|
|
||||||
kilocode)
|
|
||||||
update_agent_file "$KILOCODE_FILE" "Kilo Code"
|
|
||||||
;;
|
|
||||||
auggie)
|
|
||||||
update_agent_file "$AUGGIE_FILE" "Auggie CLI"
|
|
||||||
;;
|
|
||||||
roo)
|
|
||||||
update_agent_file "$ROO_FILE" "Roo Code"
|
|
||||||
;;
|
|
||||||
codebuddy)
|
|
||||||
update_agent_file "$CODEBUDDY_FILE" "CodeBuddy CLI"
|
|
||||||
;;
|
|
||||||
amp)
|
|
||||||
update_agent_file "$AMP_FILE" "Amp"
|
|
||||||
;;
|
|
||||||
q)
|
|
||||||
update_agent_file "$Q_FILE" "Amazon Q Developer CLI"
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
log_error "Unknown agent type '$agent_type'"
|
|
||||||
log_error "Expected: claude|gemini|copilot|cursor-agent|qwen|opencode|codex|windsurf|kilocode|auggie|roo|amp|q"
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
update_all_existing_agents() {
|
|
||||||
local found_agent=false
|
|
||||||
|
|
||||||
# Check each possible agent file and update if it exists
|
|
||||||
if [[ -f "$CLAUDE_FILE" ]]; then
|
|
||||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -f "$GEMINI_FILE" ]]; then
|
|
||||||
update_agent_file "$GEMINI_FILE" "Gemini CLI"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -f "$COPILOT_FILE" ]]; then
|
|
||||||
update_agent_file "$COPILOT_FILE" "GitHub Copilot"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -f "$CURSOR_FILE" ]]; then
|
|
||||||
update_agent_file "$CURSOR_FILE" "Cursor IDE"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -f "$QWEN_FILE" ]]; then
|
|
||||||
update_agent_file "$QWEN_FILE" "Qwen Code"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -f "$AGENTS_FILE" ]]; then
|
|
||||||
update_agent_file "$AGENTS_FILE" "Codex/opencode"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -f "$WINDSURF_FILE" ]]; then
|
|
||||||
update_agent_file "$WINDSURF_FILE" "Windsurf"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -f "$KILOCODE_FILE" ]]; then
|
|
||||||
update_agent_file "$KILOCODE_FILE" "Kilo Code"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -f "$AUGGIE_FILE" ]]; then
|
|
||||||
update_agent_file "$AUGGIE_FILE" "Auggie CLI"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -f "$ROO_FILE" ]]; then
|
|
||||||
update_agent_file "$ROO_FILE" "Roo Code"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -f "$CODEBUDDY_FILE" ]]; then
|
|
||||||
update_agent_file "$CODEBUDDY_FILE" "CodeBuddy CLI"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -f "$Q_FILE" ]]; then
|
|
||||||
update_agent_file "$Q_FILE" "Amazon Q Developer CLI"
|
|
||||||
found_agent=true
|
|
||||||
fi
|
|
||||||
|
|
||||||
# If no agent files exist, create a default Claude file
|
|
||||||
if [[ "$found_agent" == false ]]; then
|
|
||||||
log_info "No existing agent files found, creating default Claude file..."
|
|
||||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
print_summary() {
|
|
||||||
echo
|
|
||||||
log_info "Summary of changes:"
|
|
||||||
|
|
||||||
if [[ -n "$NEW_LANG" ]]; then
|
|
||||||
echo " - Added language: $NEW_LANG"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -n "$NEW_FRAMEWORK" ]]; then
|
|
||||||
echo " - Added framework: $NEW_FRAMEWORK"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]]; then
|
|
||||||
echo " - Added database: $NEW_DB"
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo
|
|
||||||
|
|
||||||
log_info "Usage: $0 [claude|gemini|copilot|cursor-agent|qwen|opencode|codex|windsurf|kilocode|auggie|codebuddy|q]"
|
|
||||||
}
|
|
||||||
|
|
||||||
#==============================================================================
|
|
||||||
# Main Execution
|
|
||||||
#==============================================================================
|
|
||||||
|
|
||||||
main() {
|
|
||||||
# Validate environment before proceeding
|
|
||||||
validate_environment
|
|
||||||
|
|
||||||
log_info "=== Updating agent context files for feature $CURRENT_BRANCH ==="
|
|
||||||
|
|
||||||
# Parse the plan file to extract project information
|
|
||||||
if ! parse_plan_data "$NEW_PLAN"; then
|
|
||||||
log_error "Failed to parse plan data"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Process based on agent type argument
|
|
||||||
local success=true
|
|
||||||
|
|
||||||
if [[ -z "$AGENT_TYPE" ]]; then
|
|
||||||
# No specific agent provided - update all existing agent files
|
|
||||||
log_info "No agent specified, updating all existing agent files..."
|
|
||||||
if ! update_all_existing_agents; then
|
|
||||||
success=false
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
# Specific agent provided - update only that agent
|
|
||||||
log_info "Updating specific agent: $AGENT_TYPE"
|
|
||||||
if ! update_specific_agent "$AGENT_TYPE"; then
|
|
||||||
success=false
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Print summary
|
|
||||||
print_summary
|
|
||||||
|
|
||||||
if [[ "$success" == true ]]; then
|
|
||||||
log_success "Agent context update completed successfully"
|
|
||||||
exit 0
|
|
||||||
else
|
|
||||||
log_error "Agent context update completed with errors"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# Execute main function if script is run directly
|
|
||||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
|
||||||
main "$@"
|
|
||||||
fi
|
|
||||||
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user