chop-dbhi / avocado / 688 / 3
80%
master: 80%

DEFAULT BRANCH: master
Ran
Files 67
Run time 176min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 81.346% (+0.1%) from 81.203%
DJANGO=1.5.12 POSTGRES_TEST_USER=postgres POSTGRES_TEST_NAME=avocado MYSQL_TEST_USER=root MYSQL_TEST_NAME=avocado

Pull #306

travis-ci

bruth 
Refactor formatter and exporter APIs to reduce overhead

A call to formatter(...) was doing a lot of work for every row being
processed. The primary slow spots are:

- Format methods being dynamically accessed via getattr() on every loop
- Unpacking keyword arguments (**kwargs) in the method call
- Building and moving OrderedDict values
- Logging a message when a value failed be coerced

To remediate these issues, targeted format methods are now processed
at initialization rather that process time. This requires the list of
formats to be passed into the constructor and not in the __call__ method.

Keyword arguments passed in during a call are no longer unpacked in the
method call, but are passed as a keyword argument itself, `kwargs`.

Use of OrderedDict has been removed altogether. These structures take up
a lot of memory and have proven to add little value during processing.
This required adding a method for getting the header/field data for the
values being processed. This is also a desirable change for addressing

Logging has been addressed by only doing while in DEBUG mode. Since the
exception is skipped, there is no point in logging the errors while in
production.

The exporters have been updated to accommodate this changes. In addition,
new read methods have been implemented for performance testing including
cached and threaded versions. The previous read method have been renamed
to manual_read since it performs manual limit and offsetting and distinct
checking.

To accommodate the new read method, the `write` on all exporters have been
changed to not call `read` itself. This change requires the passed iterable
to already contain the formatted rows.

Note: This refactor requires backwards incompatible changes to get the
performance gains.

Signed-off-by: Byron Ruth <b@devel.io>
Pull Request #306: Refactor formatter to reduce overhead

3179 of 3908 relevant lines covered (81.35%)

0.81 hits per line

Source Files on job 688.3 (DJANGO=1.5.12 POSTGRES_TEST_USER=postgres POSTGRES_TEST_NAME=avocado MYSQL_TEST_USER=root MYSQL_TEST_NAME=avocado)