jbremer / cuckoo / 119
51%
master: 62%

LAST BUILD BRANCH: expose0
DEFAULT BRANCH: master
Ran
Jobs 1
Files 161
Run time 7s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
119

push

travis-ci

jbremer 
start on reducing the thread count and thus memory footprint

Although this has been known for quite a while, the ResultServer is
quite a beast when it comes to performance and memory usage. Namely, for
every log file, behavior file, and dropped file it will create a new
thread that runs until the request has succeeded. This commit is far
from "the" solution, but it's a very small first step.

It has been noticed that, although the FileUpload and BsonParser perform
just fine'ish, the LogFile is extremely slow - in a O(n) kind of way. As
it would currently read the entire log file character-by-character,
something that allows the Python GIL [1] to move prioritization to
another thread, running multiple LogFile instances in parallel causes
all of the threads to slow down.

As an example; assuming no pauses are introduced on the client-side,
writing out just a single log file containing 1mb of characters would
take roughly 0.3 seconds. Now, running 10 of those requests in parallel
(or sequentially, really, if the sockets on the client-side are shutdown
immediately) causes great issues where - rather than finishing off a log
file every 0.3 second, *all* of the log files take ~4 seconds.

With this commit the Python processing part is at least limited to a
bare minimum, allowing for faster passthrough of log files, and as each
individual log file is written away quicker there is less chance for
clutter in other threads or the Python GIL to get the chance to slow
everything down.

[1]: https://wiki.python.org/moin/GlobalInterpreterLock

5118 of 13540 relevant lines covered (37.8%)

0.38 hits per line

Jobs
ID Job ID Ran Files Coverage
1 119.1 (ONLYINSTALL=yes) 0
37.8
 Travis Job 119.1
Source Files on build 119
Detailed source file information is not available for this build.