splitgraph / splitgraph / 1405

96%

master: 94%

push

travis-ci-com

Speed up object uploads and downloads.

Because the upload/download routine would commit/close the connection after every object, the plpy context inside the engine would reset as well, incurring a ~300ms latency importing requests/urlparse/splitgraph.config for every download (SD/GD dictionaries are cleared out too, so the modules can't be cached there). Removed that and flipped the engine into autocommit mode for the duration of the transfer.

In addition, since even `putconn()` causes this to happen, added a routine to release all non-main-thread connections back into the pool that is called at the end of object transfer.

Finally, changed object mounting to not raise an error of the object already exists (`DatabaseError` rolls back the tx state too) and bumped the connection pool size to 16 from 5 (so that downloads can happen in 15 threads).

Sample numbers:

  * Pulling `splitgraph/domestic_us_flights` (73objs, 59.17MiB): was 38s, now 20s with autocommit and pool 16.
  * Pulling `splitgraph/congress_tweets` (990objs, 470MiB): was 6m55s, now 3m3.377s (with pool 16), 2m32s (with that and autocommit).
  * Downloading 100 empty objects: was ~10s, now ~3s.

2179 of 2340 branches covered (93.12%)

Branch coverage included in aggregate %.

4788 of 4984 relevant lines covered (96.07%)

0.96 hits per line