docs

Duke-GCB · Nov 20, 2023 · 5c51dd7 · 5c51dd7
1 parent be52e6f
commit 5c51dd7
Show file tree

Hide file tree

Showing 5 changed files with 46 additions and 8 deletions.
diff --git a/conformance/ConformanceTestsJob-1.2.1.yaml b/conformance/ConformanceTestsJob-1.2.1.yaml
@@ -12,7 +12,7 @@ spec:
         command: ["chmod", "777", "/output"]
         volumeMounts:
         - mountPath: /output
-          name: conformance-output-data
+          name: conformance-test-output
       containers:
       - name: tests-1-2-1
         imagePullPolicy: IfNotPresent

diff --git a/conformance/ConformanceTestsJob-1.2.yaml b/conformance/ConformanceTestsJob-1.2.yaml
@@ -12,7 +12,7 @@ spec:
         command: ["chmod", "777", "/output"]
         volumeMounts:
         - mountPath: /output
-          name: conformance-output-data
+          name: conformance-test-output
       containers:
       - name: tests-1-2
         imagePullPolicy: IfNotPresent
@@ -53,6 +53,6 @@ spec:
         persistentVolumeClaim:
           claimName: conformance-test-data
           readOnly: true
-      - name: conformance-output-data
+      - name: conformance-test-output
         persistentVolumeClaim:
           claimName: conformance-test-output
diff --git a/conformance/Dockerfile.conformance b/conformance/Dockerfile.conformance
@@ -1,4 +1,4 @@
-FROM python:3.7
+FROM python:3.10.0-slim-buster
 LABEL maintainer="[email protected]"
 
 # cwltool requires nodejs

diff --git a/conformance/README.md b/conformance/README.md
@@ -132,25 +132,46 @@ kubectl --namespace="$NAMESPACE_NAME" create -f StageConformanceTestsData.yaml
 
 **This step can probably be simplified, but as conformance tests are in development it's too early to optimize**
 
-Calrissian does not include cwltest, so build a container that installs it with: 
+Build a container that installs `calrissian` and `cwltest` with: 
 
 ```
 minikube image build .. -t ghcr.io/calrissian/conformance:latest -f conformance/Dockerfile.conformance 
 ```
 
-This will build `calrissian:conformance` from the current source tree. You may need to tag that differently if pushing to a registry. If so, update the `image: ` in [ConformanceTestsJob-1.2.yaml](ConformanceTestsJob-1.2.yaml)
+This will build `calrissian:conformance` from the current source tree and this image is available in the minikube cluster node.
 
 ### Running Conformance Tests
 
-[ConformanceTestsJob-1.2.yaml](ConformanceTestsJob-1.2.yaml) uses `cwltest` from cwltool to run conformance tests with `--tool calrissian` and Calrissian's required arguments after `--`.
+[ConformanceTestsJob-1.2.yaml](ConformanceTestsJob-1.2.yaml) uses `cwltest` from cwltool to run conformance tests with `--tool calrissian` and Calrissian's required arguments after `--`, e.g.:
+
+```
+...
+  command: ["cwltest"]
+  args:
+  - "--test"
+  - "/conformance/common-workflow-language-1.0.2/v1.0/conformance_test_v1.0.yaml"
+  - "--tool"
+  - "calrissian"
+  - "--badgedir"
+  - "/output/badges-1.0.2"
+  - "--verbose"
+  - "--"
+  - "--max-ram"
+  - "8G"
+  - "--max-cores"
+  - "4"
+  - "--default-container"
+  - "debian:stretch-slim"
+...
+```
 
 #### CWL 1.0
 
 ```
 kubectl --namespace="$NAMESPACE_NAME" create -f ConformanceTestsJob-1.0.yaml
 kubectl --namespace="$NAMESPACE_NAME" wait --for=condition=Ready\
    --selector=job-name=conformance-tests-1-0 pods
-kubectl --namespace="$NAMESPACE_NAME" logs -f jobs/conformance-tests-1-0
+kubectl --namespace="$NAMESPACE_NAME" logs -f jobs/conformance-tests-1-0 > ../docs/conformance/1.0/result-1-0.txt
 ```
 
 #### CWL 1.1

diff --git a/docs/index.md b/docs/index.md
@@ -0,0 +1,17 @@
+# Calrissian
+
+CWL on Kubernetes
+
+## Overview
+
+Calrissian is a [CWL](https://www.commonwl.org) implementation designed to run inside a Kubernetes cluster. Its goal is to be highly efficient and scalable, taking advantage of high capacity clusters to run many steps in parallel.
+
+## Cluster Requirements
+
+Calrissian requires a [Kubernetes](https://kubernetes.io) cluster, configured to provision [PersistentVolumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) with the `ReadWriteMany` access mode. 
+
+## Scalability / Resource Requirements
+
+Calrissian is designed to issue tasks in parallel if they are independent, and thanks to Kubernetes, should be able to run very large parallel workloads.
+
+When running `calrissian`, you must provide a limit the the number of CPU cores (`--max-cores`) and RAM megabytes (`--max-ram`) to use concurrently. Calrissian will use CWL [ResourceRequirements](https://www.commonwl.org/v1.0/CommandLineTool.html#ResourceRequirement) to track usage and stay within the limits provided. We highly recommend using accurate ResourceRequirements in your workloads, so that they can be scheduled efficiently and are less likely to be terminated or refused by the cluster.