iotaledger / iota / 38664818948

72%

Build Type

push

github

Committed by web-flow

Commit Message

fix(data-ingestion-core): avoid redownloading historical files on data limit (#13171)

# Description of change

This patch raises the capacity limit checks on a per-file basis while
relaying historical checkpoints. The previous version would check on a
checkpoint basis, resulting in possible failures before the contents of
a file have been successfully sent. This caused the same file to be
downloaded again until all of its checkpoints were sent on batches
imposed by the data limit.

Now we ensure that once a file is downloaded and all of its checkpoints
collected into memory, they will be send through the channel.

In addition, the synchronization pattern for garbage collection (GC) is
changed from using an `mpsc` channel into using a `watch` channel. This
was deemed necessary to eliminate the possibility of deadlocks
introduced with the new capacity check per file.

This was due to a latent issue. Previously the executor would wait for
the reader to consume GC signals on a full channel before relaying the
read checkpoints to the workers, while the reader would wait for the
executor to read checkpoints before consuming GC signals.

## Links to any relevant issues

Fixes #13167 

## How the change has been tested

- [x] Basic tests (linting, compilation, formatting, unit/integration
tests)
- [ ] Patch-specific tests (correctness, functionality coverage)

### Infrastructure QA (only required for crates that are maintained by
@iotaledger/infrastructure)

- [x] Restart of indexer synchronization locally without resetting the
database.
- [x] Restart of indexer synchronization on a production-like database.
- [x] Deployment of services using Docker.

Tested synchronization of the first 200k checkoints after epoch 652 on
`testnet` against the `develop` version, for an indexer instance running
on default configuration. The patch is found to be 9 times faster than
the `develop` version.

### Release Notes

- [x] Indexer: :screwdriver: Fixes an issue where the ... (continued)

Coverage Stats

10 of 11 new or added lines in 2 files covered. (90.91%)

216 existing lines in 12 files now uncovered.

188415 of 266380 relevant lines covered (70.73%)

447886.61 hits per line