1
99%
master: 99%

Ran 02 Sep 2015 03:40AM UTC

Files 121

Run time 3s

Badge

Embed ▾

Committed 02 Sep 2015 03:35AM UTC coverage: 99.427%. Remained the same

Job # 2.2.2

Build Type

push

travis-ci

Committed by

monfresh

Commit Message

Use SmarterCSV to optimize import script

The import script was consuming a lot of memory because it was reading
the CSV files with `File.read`, then loading the contents in an array
with `CVS.new.entries`.

The `File.read` part is not necessary since you can pass in a path
directly to `CSV.new` or `SmarterCSV.process`. SmarterCSV also provides
a way to load the contents in chunks to consume less memory.

These changes will help with memory consumption, but not speed.
The script takes so long because of the various `touch`es that are
required to populate the full-text search index.

The easiest way to speed up the import is to manually remove those
callbacks temporarily, then update the search index manually. On a
sample data set, this brought the total time down to 11 minutes from
31 minutes.

I've updated the installation instructions with these guidelines for
speeding up the script.

Run Details

1735 of 1745 relevant lines covered (99.43%)

66.43 hits per line

codeforamerica / ohana-api / 1062 / 1
99%
master: 99%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 1062.1 (2.2.2)

codeforamerica / ohana-api / 1062 / 1 99% master: 99%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 1062.1 (2.2.2)

codeforamerica / ohana-api / 1062 / 1
99%
master: 99%

README BADGES
x