iotaledger / iota / 17365174154

push

github

feat(starfish): improving transaction synchronizer (#8157)

# Description of change

In this PR, we improve the transaction synchronizer in Starfish. Main
changes:
- we join fetching and processing transactions into one method
`fetch_and_process_transactions_from_authorities` to reuse this for live
and periodic syncing
- we make at most one fetch request to a given authority in one call of
`fetch_and_process_transactions_from_authorities`
- we track the number of concurrent requests to each authority and limit
that value using a new struct in synchronizer called `active_requests`
- we process transactions after fetching for each authority
individually. While this creates much more calls to the core thread,
this improves the syncing process significantly since it was slowed down
by the slowest response in the previous implementation. Later on we can
add one/a few active thread(s) that will take all fetched bytes and
process them, removing duplicates and sending processed transactions to
the core thread on a regular basis.
- we changed some parameters for fetching transactions, but they might
require further changes once the protocol is tested on a larger network
with longer discrepancies.

## Links to any relevant issues

Fixes #8147 

## How the change has been tested

- [x] Basic tests (linting, compilation, formatting, unit/integration
tests)
- [x] Patch-specific tests (correctness, functionality coverage)
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have checked that new and existing unit tests pass locally with
my changes

### Release Notes

- [x] Protocol: Improvements in Starfish tx synchronizer
- [ ] Nodes (Validators and Full nodes):
- [ ] Indexer:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [ ] REST API:

359 of 372 new or added lines in 4 files covered. (96.51%)

228828 of 347474 relevant lines covered (65.85%)

2273351.75 hits per line