rlgomes / flume
96%
master: 93%

LAST BUILD BRANCH: dependabot/pip/urllib3-1.26.5
DEFAULT BRANCH: master
Repo Added
Files 65
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH optimize-various-reducers-in-elastic-adapter
branch: optimize-various-reducers-in-elastic-adapter
CHANGE BRANCH
x

71

push

travis-ci

rlgomes 
optimized count, maximum, minimum in es adapter

fixes #39

with this we'll have a tremendous performance boost since since these
reductions are done in the storage layer and we don't need to pull
thousands of data points all the way into the **flume** runtime engine
in order to calculate the same things.

As an example of the performance boost:

```bash
rlgomes@t460s> flume "emit(limit=100000, start='1970-01-01') |
put(count=count()) | write('elastic', index='test', batch=1024)"
2016-09-06 21:39:50 $?=0 pwd=/home/rlgomes/workspace/python/flume
venv=env branch=(no duration=38.256s
rlgomes@t460s> flume --no-optimize "read('elastic', index='test') |
reduce(count=count()) | write('stdio')"
{"count": 100000, "time": "1970-01-01T00:00:00.000Z"}
2016-09-06 21:41:26 $?=0 pwd=/home/rlgomes/workspace/python/flume
venv=env branch=(no duration=72.940s
rlgomes@t460s> flume "read('elastic', index='test') |
reduce(count=count()) | write('stdio')"
{"count": 100000, "time": "1970-01-01T00:00:00.000Z"}
2016-09-06 21:42:07 $?=0 pwd=/home/rlgomes/workspace/python/flume
venv=env branch=(no duration=.473s
rlgomes@t460s> flume --no-optimize "read('elastic', index='test') |
reduce(max=maximum('count')) | write('stdio')"
{"max": 100000, "time": "1970-01-01T00:00:00.000Z"}
2016-09-06 21:43:21 $?=0 pwd=/home/rlgomes/workspace/python/flume
venv=env branch=(no duration=57.814s
rlgomes@t460s> flume "read('elastic', index='test') |
reduce(max=maximum('count')) | write('stdio')"
{"max": 100000.0, "time": "1970-01-01T00:00:00.000Z"}
2016-09-06 21:44:17 $?=0 pwd=/home/rlgomes/workspace/python/flume
venv=env branch=(no duration=.439s
rlgomes@t460s> flume --no-optimize "read('elastic', index='test') |
reduce(min=minimum('count')) | write('stdio')"
{"time": "1970-01-01T00:00:00.000Z", "min": 1}
2016-09-06 21:45:16 $?=0 pwd=/home/rlgomes/workspace/python/flume
venv=env branch=(no duration=44.500s
rlgomes@t460s> flume "read('elastic', index='test') |
reduce(min=minimum('count')) | write('... (continued)

1810 of 1895 relevant lines covered (95.51%)

1.9 hits per line

Relevant lines Covered
1895 RELEVANT LINES 1810 COVERED LINES
1.9 HITS PER LINE
Source Files on optimize-various-reducers-in-elastic-adapter
Coverage ∆ File Lines Relevant Covered Missed Hits/Line

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
71 optimize-various-reducers-in-elastic-adapter optimized count, maximum, minimum in es adapter fixes #39 with this we'll have a tremendous performance boost since since these reductions are done in the storage layer and we don't need to pull thousands of data points all the way into the **fl... push rlgomes travis-ci pending completion  
See All Builds (81)