MITLibraries / slingshot

83%

master: 93%

LAST BUILD ON BRANCH multipart

branch: multipart

CHANGE BRANCH

x

Reset

• multipart
• 0.2.1
• 0.2.2
• 0.3.0
• 0.3.1
• 2to3
• add-download
• adr
• auth
• aws
• bag_dir
• batch-solr
• boto-session
• buffered-io
• bumpversion
• cli
• cli-docs
• db-options
• dependabot/pip/cryptography-3.2
• dependabot/pip/ecdsa-0.13.3
• dependabot/pip/jinja2-2.11.3
• dependabot/pip/py-1.10.0
• dependabot/pip/pyyaml-5.4
• dependabot/pip/rsa-4.1
• dependabot/pip/rsa-4.7
• deploy
• deps
• docker
• dockerignore
• docs
• exception-bug
• failure-#9
• fgdc-acl
• fgdc-keywords
• fgdc-test
• fgdc-title
• fgdc-uri
• fgdc-xml
• fix-content-type
• fix-quoting
• fix-region
• flake8
• flatten-zip
• flush-logs
• format-exc
• geoserver-adr
• geoserver-auth
• geotiff
• image-type
• init-geoserver
• int-tests
• issue-#13
• logging-#8
• marc
• marc-status
• master
• metadata
• mf-changes
• minio
• multiple-layers
• namespace
• no-tiff
• ogc-url
• overwrite-bag
• pg-load
• pg-shapereader
• pipenv-changes
• pipenv-script
• pipenv-sys
• pipfile
• publish-all
• python-version
• raster
• record-refactor
• reindex
• rel
• rel-0.4
• rel-0.5
• rel-0.7.0
• rel-0.8
• remove-ogre
• remove-reqmts
• rename-record
• s3
• s3-marc
• schema
• security
• shapereader
• single-layer-pub
• skip-existing
• skip-loaded
• solr-commit
• streamzip
• threadsafe-session
• tiff-dir
• tiff-url
• traceback
• uni
• update-deps
• update-version
• upgrade-pip
• use-json
• uuid-unicode
• v0.2.0
• v0.2.1
• v0.2.2
• v0.3.0
• v0.3.1
• v0.4.0
• v0.5.0
• v0.5.1
• v0.6.0
• v0.7.0
• v0.8.0
• v0.9
• v0.9.0
• zipped-submission

Build # 355

Build Type

Pull #104

travis-ci

Committed by web-flow

Commit Message

Fix S3 read/write performance issues

The S3IO wrapper class that was needed to make an S3 object function
like a file led to some pretty bad performance problems. An object was
read in most cases at no more than 8192 bytes at a time. As you can
imagine, this is not an efficient way to read a large file over the
network.

These changes switch to using multipart uploads for S3 and manually
request chunks of the file at a specified chunk size when reading the
zipfile. The Wrapper class that inherits from BufferedIOBase is a shim
for loading the shapefile into Postgres because it's not possible to
control the buffering otherwise.

The S3_BUFFER_SIZE constant controls how much is requested from S3 at a
time. Keep in mind you will likely see an increase in memory consumption
equal to roughly this number times however many worker threads you use.

It's probably worth revisiting the whole IO system at some point. The
biggest constraint probably comes from running this on Fargate.
Exploring other deployment options may make this easier in the future.

Pull Request Pull Request #104: Fix S3 read/write performance issues

Run Details

48 of 48 new or added lines in 4 files covered. (100.0%)

650 of 787 relevant lines covered (82.59%)

0.83 hits per line