MITLibraries / slingshot / 355

83%

master: 93%

Pull #104

travis-ci

Fix S3 read/write performance issues

The S3IO wrapper class that was needed to make an S3 object function
like a file led to some pretty bad performance problems. An object was
read in most cases at no more than 8192 bytes at a time. As you can
imagine, this is not an efficient way to read a large file over the
network.

These changes switch to using multipart uploads for S3 and manually
request chunks of the file at a specified chunk size when reading the
zipfile. The Wrapper class that inherits from BufferedIOBase is a shim
for loading the shapefile into Postgres because it's not possible to
control the buffering otherwise.

The S3_BUFFER_SIZE constant controls how much is requested from S3 at a
time. Keep in mind you will likely see an increase in memory consumption
equal to roughly this number times however many worker threads you use.

It's probably worth revisiting the whole IO system at some point. The
biggest constraint probably comes from running this on Fargate.
Exploring other deployment options may make this easier in the future.

48 of 48 new or added lines in 4 files covered. (100.0%)

650 of 787 relevant lines covered (82.59%)

0.83 hits per line