isislovecruft / bridgedb / 628
72%
develop: 91%

LAST BUILD BRANCH: bridgedb-0.6.5
DEFAULT BRANCH: develop
Ran
Jobs 2
Files 32
Run time 4min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
628

push

travis-ci

isislovecruft 
"All scalability problem can be solve by bolt a message queue on top."
                                              ― @BigDataBorat

Create Redis transaction queues for handing large numbers of descriptors.

This is a gross hack. However, no matter what I seem to do, a running
Redis server (not run as root, and without doing `sudo ulimit -c` to
change the number of available file descriptors) will not handle storing
more than 1000 requests to store serialised descriptors at a time.

I know that it is a bad design choice to "bolt a message queue on top"
to handle scalability issues. [0] But I don't want to require root to
start the database, since Redis appears to have no mechanism for
dropping privileges, nor for setting the file descriptor limit (as
little-t tor does).

In my first iteration of this, I created one RedisClient per
transaction. The code for that is conceptually simpler, and much
cleaner. However, for parsing and storing 250 bridge networkstatus
descriptors, the amount of memory being used due to the perpetual
initialisation and destruction of the RedisClient classes was going
through the roof (hooray, Python…), and the mean time for the parsing
and the set of transactions was 62 milliseconds.

Creating only one RedisClient is conceptually more jumbled, due to the
arrangements of the Twisted callback/errback chains and the need for the
creation of a twisted.internet.defer.DeferredList. However, memory
requirements were acceptable, and the mean time for parsing and storing
250 bridge networkstatus descriptors dropped to 12 milliseconds.

However, if we try to use only one RedisClient, and we happen to get a
file from the BridgeAuthority with greater than 1000 descriptors in it,
the Redis server will begin arbitrarily killing connections due to the
ulimit.

Hence, my ugly hack.

 * TODO: separate the message queue and redis worker creating logic into
   a separate class, or set of functions, so that it works for any set
   of transaction... (continued)

2894 of 4036 relevant lines covered (71.7%)

1.43 hits per line

Jobs
ID Job ID Ran Files Coverage
1 628.1 (TWISTED_VERSION=13.2.0 PYOPENSSL_VERSION=0.13.1) 0
71.7
 Travis Job 628.1
3 628.3 (TWISTED_VERSION=14.0.0 PYOPENSSL_VERSION=0.14) 0
71.68
 Travis Job 628.3
Source Files on build 628
Detailed source file information is not available for this build.