workflow.txt

Admin interface monitors/controls the system:

== Blank slate
 -- empty corpus
 -- empty 0 x 0 job landscape

== Initialization
 1. Register a corpus/resource
 2. Register services ("blackboard modules") per corpus!

== Job Queue

We have an M x N x C job landscape, for M services and N corpus entries in C corpora.
Job status Codes:
-n or less: Blocked by (n-5) prerequisite services 
-5 ready for processing
-4 Done with Fatal errors
-3 Done with Errors
-2 Done with Warnings
-1 Done OK
JOBID > 0 incomplete/processing

== Daemonized processing
 1. If we have space in our job queue 
  1.1. Grab next bundle of ready to process jobs or sleep.
  1.2. Pass jobs on to Gearman for work aynchronously
 2. Once job bundle is completed, register results in TaskDB, DocDB and MetaDB.

Gearman could in fact take charge of most of the cron problems -- just hit it with jobs.

== Third-party service registrations
 - validate all mandatory fields, good UI
 - have a mandatory testbed sandbox - one article from each corpus? total 5?
  the developer can only deploy on an entire corpus once the test bed has finished with non-error results (5 out of 5)
  and the developer manually confirms they are happy with the annotations.

== Dependency workflows
 Addition workflow
 - When adding a new task S1
 - set generic status = -5 - #foundations
 - For each entry, each foundation, if status = -1 or -2, +1 on S1 status. (already completed foundations can be directly used)

Deletion workflow - will not implement, inconsistent states will be hard to manage.
 (the straightforward solution would be deletion propagation to all dependent services)
# - When deleting a task S1
# - For every master having S1 as foundation, on every entry which is currently blocked, # or queued, -1 on the status (block until S1 completes successfully).

 Completion workflow
 - on task completion with status OK or Warning, enable +1 on status of master service 
 for the same entry (if currently blocked).
 
 Rerun workflow  = one can only rerun COMPLETED tasks
 (this is also the Service Update workflow)
 - When rerunning service S1
 - Set generic status = -5 - #foundations
 - For each entry, each foundation, if status = -1 or -2, +1 on S1 status. (already completed foundations can be directly used)
 - Conversely, for every master having S1 as foundation, recursively propagate a rerun request