tethysplatform / tethys / 31835103680

100%

push

github

Let job status be reported to the portal instead of polled for (#1294)

* update to latest version of thredds image 5.4 is no longer available

* Let job status be reported to the portal instead of polled for

The jobs table poll views refresh job status on every request. For condor
jobs that opens an SSH connection to the scheduler inside the
request/response cycle, and rendering a workflow's DAG costs one remote
query per node -- measured at ~115 ms per expanded node, so a 26-node DAG
spends around 6 s per poll after the cheaper wins in condorpy, and 26 s
before them.

Two halves, which are only useful together.

Serve from the database:

- CondorWorkflowNode.cached_node_status holds each node's last known condor
  status. apply_node_statuses() persists a mapping of them, and
  update_node_statuses() obtains one with a single query for the whole DAG
  and hands it to the same method.
- CondorWorkflow.node_statuses_updated records when that last happened, and
  node_statuses_are_current reports whether it falls within
  node_statuses_max_age. The poll views render the DAG from the database
  when it does, and read live statuses when it does not.
- CondorWorkflow.cached_statuses reports node status counts from the
  persisted values in the same form as CondorBase.statuses.

Accept reports rather than only asking:

- report_job_status() takes a status, and optionally node statuses, from
  whoever already knows them. update_job_status() asks the job's source what
  the status is, which for condor means the portal SSHes out to rediscover
  what it was just told; this records what it is given.
- Reports are authorised by TethysJob.status_report_token, signed with the
  portal's SECRET_KEY. Nothing has to be configured and no secret has to be
  distributed, and a token authorises reports for one job and no other.
- A terminal status is recorded even when the work that follows it fails, so
  a transient problem syncing results does not leave the portal beli... (continued)

146 of 146 new or added lines in 5 files covered. (100.0%)

13867 of 13868 relevant lines covered (99.99%)

1.0 hits per line