gnosis / dex-services / 4509

31%

push

travis-pro

Parallelize batch processing in E2E scripts (#1229)

This PR uses the `rayon` library to parallelize processing each batch to speed things up. This takes advantage of the fact that the auction state and `pricegraph` can be computed "on-the fly" from an immutable event collection in order to take full advantage of all CPU cores :rocket: 

Additionally, the nasty threading bits and dealing with synchronization has been abstracted behind a `Reporting` trait to make things easier and reduce boiler plate.

:warning: There is one disadvantage in that the output is **no longer in order**. I don't think that is a huge issue, since the data can be ordered when it is being processed and analyzed.

### Test Plan

Before `233.85s` :turtle: 
```
$ time -p target/release/historic_prices --orderbook-file target/orderbook.mainnet.bin
54678 / 54678 [==============================================] 100.00 % 230.39/s
Processed 13743 token prices: bad estimates 45.19%, average error 27643.61%.
real 237.53
user 233.85
sys 2.78
```

After `53.27s` :rocket: 
```
$ time -p target/release/historic_prices --orderbook-file target/orderbook.mainnet.bin
54678 / 54678 [=============================================] 100.00 % 1033.13/s
Processed 13743 token prices: bad estimates 45.22%, average error 27938.76%.
real 53.27
user 397.93
sys 2.05
```

A 4.5x increase on an 8-core machine. Notice how the `user` time is almost double. We are losing a lot of time dealing with thread synchronization, and might be able to speed it up even a bit more.

11572 of 46233 branches covered (25.03%)

Branch coverage included in aggregate %.

3072 of 4918 relevant lines covered (62.46%)

12359.91 hits per line