thriftrw / thriftrw-python

74%

master: 92%

LAST BUILD ON BRANCH streaming

branch: streaming

CHANGE BRANCH

x

Reset

• streaming
• 1.2.0
• 1.2.1
• 1.2.2
• 1.2.3
• 1.2.4
• 1.2.5
• 1.3.0
• 1.4.0
• 1.5.0
• 1.5.1
• 1.6.0
• 1.7.0
• 1.7.1
• 1.7.2
• 1.8.0
• 1.8.1
• abg/fix-build
• abg/input-too-short
• abg/response-from-the-future
• abhinav-patch-1
• benchmark-underp
• chen/profile
• decimalfix
• enum-from-the-future
• enum-item-duplicates
• fixbuild
• hash
• license
• master
• message
• more-bench
• mutable-defaults
• no-malloc
• non-strict-exceptions
• opt2
• optread
• release-1.6
• saveopt2
• setup-dont-check-version
• streaming-dev
• streaming-read
• strfix
• struct-docfix
• testmasterbuild
• travis-migrate
• validation-opt

Build # 614

Build Type

push

travis-ci

Committed by web-flow

Commit Message

Streaming read optimization. (#136)

This PR refactors deserialization to read directly from memory. That is, the serialization/deserialization path used to allocate intermediate representations called "Values" before doing a second pass that converted the values into primitive python objects. This process introduces considerable overhead however. By removing this layer and deserializing directly into python objects, we can dramatically improve performance.

Benchmarks:

Before:

--------------------------------------------------------------------- benchmark: 8 tests ---------------------------------------------------------------------
Name (time in us)                                      Min           Max          Mean      StdDev        Median          IQR  Outliers(*)  Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------
test_binary_loads[nested_structs]              16,474.0086   33,906.9366   20,297.8651  5,972.9826   17,036.4380   2,691.3881        11;11      48           1
test_binary_loads[primitive_containers]       352,126.8368  374,081.8501  363,660.1448  8,221.5467  362,915.9927  10,945.7970          2;0       5           1
After:

--------------------------------------------------------------------- benchmark: 8 tests ---------------------------------------------------------------------
Name (time in us)                                      Min           Max          Mean      StdDev        Median          IQR  Outliers(*)  Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------
test_binary_loads[nested_structs]              10,932.9224   29,603.0045   12,143.8092  2,779.9176   11,502.9812    415.9808          3;9      87           1
test_binary_loads[primitive_containers]      ... (continued)

Run Details

2191 of 2947 relevant lines covered (74.35%)

0.74 hits per line