4
83%
master: 93%

Ran 17 Jul 2019 05:04PM UTC

Files 10

Run time 2s

Badge

Embed ▾

Committed 17 Jul 2019 05:01PM UTC coverage: 82.592% (-1.0%) from 83.58%

Job # TOX_ENV=coveralls AWS_SECRET_ACCESS_KEY=secret AWS_ACCESS_KEY_ID=key

Build Type

Pull #104

travis-ci

Committed by

web-flow

Commit Message

Fix S3 read/write performance issues

The S3IO wrapper class that was needed to make an S3 object function
like a file led to some pretty bad performance problems. An object was
read in most cases at no more than 8192 bytes at a time. As you can
imagine, this is not an efficient way to read a large file over the
network.

These changes switch to using multipart uploads for S3 and manually
request chunks of the file at a specified chunk size when reading the
zipfile. The Wrapper class that inherits from BufferedIOBase is a shim
for loading the shapefile into Postgres because it's not possible to
control the buffering otherwise.

The S3_BUFFER_SIZE constant controls how much is requested from S3 at a
time. Keep in mind you will likely see an increase in memory consumption
equal to roughly this number times however many worker threads you use.

It's probably worth revisiting the whole IO system at some point. The
biggest constraint probably comes from running this on Fargate.
Exploring other deployment options may make this easier in the future.

Pull Request Pull Request #104: Fix S3 read/write performance issues

Run Details

650 of 787 relevant lines covered (82.59%)

0.83 hits per line

MITLibraries / slingshot / 355 / 4
83%
master: 93%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 355.4 (TOX_ENV=coveralls AWS_SECRET_ACCESS_KEY=secret AWS_ACCESS_KEY_ID=key)

MITLibraries / slingshot / 355 / 4 83% master: 93%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 355.4 (TOX_ENV=coveralls AWS_SECRET_ACCESS_KEY=secret AWS_ACCESS_KEY_ID=key)

MITLibraries / slingshot / 355 / 4
83%
master: 93%

README BADGES
x