vanvalenlab / kiosk-redis-consumer / 319

96%

push

travis-ci

Update the Redis client wrapper with Sentinel support (#43)

* use redis sentinel to route requests to slaves or masters.

* use LLEN instead of RPOPLPUSH to see if the queue is empty.

* add summary data to redis consumer

* retry BUSY redis ResponseErrors

* convert get_image_files_from_dir to iterator to avoid extracting all files at once

* retry all gcloud network errors

* call hmget instead of hgetall (O(5) instead of O(N))

* save total_seconds in the summary stats of an image job

* DRY out hvals.get(children)

* ZipFileConsumer uses its own queue-zip queue name.

* add data-processing functionality to redis-consumer

* create the client inside each retry loop

* add InternalServerError to the retry loop for GoogleStorage

* update _put_back_hash to only call RPOPLPUSH to avoid keys getting lost

* update storage tests for client re-creation (avoid infinite loop)

* zip consumer should update the key each time it deals with a child file

* add configurable EXPIRE_TIME env var

* Move all items in the processing queue to the work queue at the start of consume()

* ZipConsumer expires all child keys AFTER it has successfully updated the status.

* on ConnectionError, update the master and slave nodes.

* close grpc channels when request is finished.

* if setting status to cleanup, just do the cleanup, no point in putting it back.

* workaround for missing output_file_name bug

253 of 253 new or added lines in 6 files covered. (100.0%)

783 of 969 relevant lines covered (80.8%)

2.42 hits per line