isislovecruft / bridgedb / 628 / 1
72%
develop: 91%

LAST BUILD BRANCH: bridgedb-0.6.5
DEFAULT BRANCH: develop
Ran
Files 32
Run time 15s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 71.705%. Remained the same
TWISTED_VERSION=13.2.0 PYOPENSSL_VERSION=0.13.1

push

travis-ci

isislovecruft 
"All scalability problem can be solve by bolt a message queue on top."
                                              ― @BigDataBorat

Create Redis transaction queues for handing large numbers of descriptors.

This is a gross hack. However, no matter what I seem to do, a running
Redis server (not run as root, and without doing `sudo ulimit -c` to
change the number of available file descriptors) will not handle storing
more than 1000 requests to store serialised descriptors at a time.

I know that it is a bad design choice to "bolt a message queue on top"
to handle scalability issues. [0] But I don't want to require root to
start the database, since Redis appears to have no mechanism for
dropping privileges, nor for setting the file descriptor limit (as
little-t tor does).

In my first iteration of this, I created one RedisClient per
transaction. The code for that is conceptually simpler, and much
cleaner. However, for parsing and storing 250 bridge networkstatus
descriptors, the amount of memory being used due to the perpetual
initialisation and destruction of the RedisClient classes was going
through the roof (hooray, Python…), and the mean time for the parsing
and the set of transactions was 62 milliseconds.

Creating only one RedisClient is conceptually more jumbled, due to the
arrangements of the Twisted callback/errback chains and the need for the
creation of a twisted.internet.defer.DeferredList. However, memory
requirements were acceptable, and the mean time for parsing and storing
250 bridge networkstatus descriptors dropped to 12 milliseconds.

However, if we try to use only one RedisClient, and we happen to get a
file from the BridgeAuthority with greater than 1000 descriptors in it,
the Redis server will begin arbitrarily killing connections due to the
ulimit.

Hence, my ugly hack.

 * TODO: separate the message queue and redis worker creating logic into
   a separate class, or set of functions, so that it works for any set
   of transaction... (continued)

2894 of 4036 relevant lines covered (71.7%)

0.72 hits per line

Source Files on job 628.1 (TWISTED_VERSION=13.2.0 PYOPENSSL_VERSION=0.13.1)