codeforamerica / ohana-api / 1060

99%

master: 99%

push

travis-ci

Use SmarterCSV to optimize import script

The import script was consuming a lot of memory because it was reading
the CSV files with `File.read`, then loading the contents in an array
with `CVS.new.entries`.

The `File.read` part is not necessary since you can pass in a path
directly to `CSV.new` or `SmarterCSV.process`. SmarterCSV also provides
a way to load the contents in chunks to consume less memory.

These changes will help with memory consumption, but not speed.
The script takes so long because of the various `touch`es that are
required to populate the full-text search index.

The easiest way to speed up the import is to manually remove those
callbacks temporarily, then update the search index manually. On a
sample data set, this brought the total time down to 11 minutes from
31 minutes.

I've updated the installation instructions with these guidelines for
speeding up the script.

43 of 43 new or added lines in 14 files covered. (100.0%)

1735 of 1745 relevant lines covered (99.43%)

66.43 hits per line