Merge remote-tracking branch 'origin/test' into prod; JIRA: PGAPX-473…

…; PGAPX-480; PGAPX-497; PGAPX-488

some of the commits that went into (generic automatically generated 'merge' messages omitted)

remove passing bacterial_screening/feats to screen_evaluate: we already analyzed it in fscr_calls_pass1, no need to add unfiltered input from earlier in processing; JIRA: PGAPX-473
wrapper for fastaval created, parameter editing tested; JIRA: PGAPX-473
switch to using fastaval.sh wrapper, temporarily, and pass ignore_all_errors to it, permanently; JIRA: PGAPX-473
moved fastaval.sh from github repo here; JIRA: PGAPX-473
switching to sparclbl_mp.sh and added coresMin=4 requirement; JIRA: PGAPX-480
conflicts resolved

coresMin=1 for the sake of most barebone Mac users; JIRA: PGAPX-480
remove coreMin spec as per standup decision; JIRA: PGAPX-480
sleep only when the queue is empty; JIRA: PGAPX-497
added some commentary on what we are doing while channeling output from a subprocess; JIRA: PGAPX-497
more elegant way of doing the timeout; JIRA: PGAPX-488

pgap.cwl

@@ -113,6 +113,7 @@ steps:
             default: 200
         check_internal_ns:
             default: true
+        ignore_all_errors: ignore_all_errors
     out: [success]
   vecscreen:
         run: vecscreen/vecscreen.cwl

progs/fastaval.cwl

@@ -1,6 +1,6 @@
 cwlVersion: v1.0
 class: CommandLineTool
-baseCommand: fastaval
+baseCommand: fastaval.sh
 label: fastaval
 requirements:
   - class: InlineJavascriptRequirement
@@ -45,6 +45,8 @@ inputs:
     inputBinding:
       prefix: '-check_internal_ns'
     doc: Check for internal N's
+  ignore_all_errors:
+        type: boolean?
 outputs:
   out:
         type: File

progs/sparclbl.cwl

@@ -2,7 +2,11 @@ cwlVersion: v1.0
 label: "sparclbl"
 
 class: CommandLineTool
-baseCommand: sparclbl.sh
+requirements:
+ - class: ResourceRequirement
+   ramMax: 15000
+
+baseCommand: sparclbl_mp.sh
 inputs:
   s: 
     type: File? # to accomodate output of asn2fasta which is of type [null, File]

scripts/pgap.py

@@ -258,16 +258,24 @@ def output_reader(proc, outq):
                     for line in fIn:
                         f.write(line)
                 f.write("--- End YAML Input ---\n")
+
+                # Check if child process has terminated
                 while proc.poll() == None:
+
+                    # this loop is to avoid (expensive?) checking if process 'proc' terminated
                     while True:
                         try:
-                            line = outq.get(block=False)
+
+                            # either process terminated or it is "thinking"
+                            # let's wait a little bit before checking termination:
+                            line = outq.get(timeout=0.1)
                             f.write(line)
                             if (self.params.args.verbose) or pat.match(line):
                                 print(line, end='')
                         except queue.Empty:
+
+                            # go to outer while loop to check proc.poll() == None
                             break
-                        time.sleep(0.1)
 
         finally:
             proc.terminate()

vecscreen/vecscreen.cwl

@@ -52,7 +52,6 @@ steps:
   screen_evaluate:
         run: ../progs/screen_evaluate.cwl
         in:
-            asn: bacterial_screening/feats
             tab: FSCR_Calls_first_pass/calls
             ifmt:
                 default: seq-annot