chop-dbhi / avocado
81%
master: 80%

LAST BUILD BRANCH: instance-cache-key
DEFAULT BRANCH: master
Repo Added
Files 69
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH refactor-export
branch: refactor-export
CHANGE BRANCH
x

690

push

travis-ci

bruth 
Refactor formatter and exporter APIs to reduce overhead

A call to formatter(...) was doing a lot of work for every row being
processed. The primary slow spots are:

- Format methods being dynamically accessed via getattr() on every loop
- Unpacking keyword arguments (**kwargs) in the method call
- Building and moving OrderedDict values
- Logging a message when a value failed be coerced

To remediate these issues, targeted format methods are now processed
at initialization rather that process time. This requires the list of
formats to be passed into the constructor and not in the __call__ method.

Keyword arguments passed in during a call are no longer unpacked in the
method call, but are passed as a keyword argument itself, `kwargs`.

Use of OrderedDict has been removed in favor of a lightweight namedtuple
class as input (for named values) and plain tuples for output. Value
fields are now defined in the `get_*_header` methods. A custom method
can be implemented per format type.

Exceptions that occur during formatting are no long silently skipped
unless the `ExpectedFormatterException` is used.

The exporters have been updated to accommodate this changes. In addition,
new read methods have been implemented for performance testing including
cached and threaded versions. The previous read method have been renamed
to manual_read since it performs manual limit and offsetting and distinct
checking.

To accommodate the new read method, the `write` on all exporters have been
changed to not call `read` itself. This change requires the passed iterable
to already contain the formatted rows.

Note: This refactor requires backwards incompatible changes to get the
performance gains.

Signed-off-by: Byron Ruth <b@devel.io>

3172 of 3893 relevant lines covered (81.48%)

2.44 hits per line

Relevant lines Covered
3893 RELEVANT LINES 3172 COVERED LINES
2.44 HITS PER LINE
Source Files on refactor-export
Coverage ∆ File Lines Relevant Covered Missed Hits/Line

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
690 refactor-export Refactor formatter and exporter APIs to reduce overhead A call to formatter(...) was doing a lot of work for every row being processed. The primary slow spots are: - Format methods being dynamically accessed via getattr() on every loop - Unpacki... push bruth travis-ci pending completion  
687 refactor-export Refactor formatter and exporter APIs to reduce overhead A call to formatter(...) was doing a lot of work for every row being processed. The primary slow spots are: - Format methods being dynamically accessed via getattr() on every loop - Unpacki... push bruth travis-ci pending completion  
686 refactor-export Fix (new) flake8 issues Signed-off-by: Byron Ruth <b@devel.io> push bruth travis-ci pending completion  
672 refactor-export Refactor formatter to reduce overhead A call to formatter(...) was doing a lot of work for every row being processed. The primary slow spots are: - Format methods being dynamically accessed via getattr() on every loop - Unpacking keyword argumen... push bruth travis-ci pending completion  
671 refactor-export Refactor formatter to reduce overhead A call to formatter(...) was doing a lot of work for every row being processed. The primary slow spots are: - Format methods being dynamically accessed via getattr() on every loop - Unpacking keyword argumen... push bruth travis-ci pending completion  
670 refactor-export Refactor formatter to reduce overhead A call to formatter(...) was doing a lot of work for every row being processed. The primary slow spots are: - Format methods being dynamically accessed via getattr() on every loop - Unpacking keyword argumen... push bruth travis-ci pending completion  
See All Builds (164)