iotaledger / iota / 40983832694
72%

DEFAULT BRANCH: develop
Ran
Jobs 1
Files 1140
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 72.314% (+0.5%) from 71.826%
40983832694

push

github

web-flow 
feat(iota-core,iota-config,iota-json-rpc): keep history in per-epoch buckets and merge the RPC index stores (#13377)

# Description of change

Merges the storage pruning stack, reviewed and squash-merged PR by PR.

- One `rpc_indexes` store replaces the JSON-RPC and gRPC index stores.
It is indexed per checkpoint and rebuilt on formal-snapshot restore.
- Superseded object versions, executed transactions, checkpoints and the
index history are kept in per-epoch buckets, and pruning drops whole
expired epochs.
- The pruning settings count epochs, each kind of data has its own
retention, and the per-epoch history can be given its own directory.
- The first start migrates the existing history into buckets, using SST
ingestion and a parallel sweep.
- Transactions are reported as final only once their checkpoint is
indexed, so reads right after execution see them.

## Links to any relevant issues

Merged PRs, in order:
- #12332
- #12637
- #12813
- #12645
- #12652
- #12693
- #12695
- #12737
- #12762
- #12780
- #12837
- #12841
- #13114
- #13160
- #13172
- #13173
- #13216
- #13231
- #13233
- #13234
- #13235
- #13387
- #13380
- #13384

## How the change has been tested

Each PR was tested and reviewed on its own; see the linked PRs.

10148 of 11281 new or added lines in 89 files covered. (89.96%)

732 existing lines in 53 files now uncovered.

202552 of 280100 relevant lines covered (72.31%)

444399.87 hits per line

Uncovered Changes

Lines Coverage ∆ File
159
86.8
crates/iota-core/src/rpc_indexes/mod.rs
110
88.26
crates/iota-core/src/rpc_indexes/jsonrpc_api.rs
87
35.05
35.05% crates/iota-tool/src/db_tool/index_search.rs
61
18.68
18.68% crates/iota-tool/src/db_tool/mod.rs
54
94.47
crates/iota-core/src/authority/ledger_backlog_migration.rs
46
85.32
0.77% crates/iota-core/src/checkpoints/mod.rs
43
87.78
0.59% crates/iota-core/src/authority.rs
40
0.0
0.0% crates/iota-node/src/main.rs
36
86.91
crates/iota-core/src/rpc_indexes/live_scan.rs
34
92.29
crates/iota-core/src/authority/historic_ledger.rs
29
0.0
0.0% crates/iota-tool/src/lib.rs
28
96.85
crates/iota-core/src/epoch_buckets.rs
28
44.69
3.2% crates/iota-tool/src/db_tool/db_dump.rs
27
92.64
5.81% crates/iota-core/src/authority/authority_store_tables.rs
25
84.08
crates/iota-core/src/checkpoints/historic_checkpoints.rs
22
78.47
6.85% crates/iota-core/src/authority/authority_store.rs
22
90.69
0.81% crates/iota-node/src/lib.rs
22
69.94
0.25% crates/simulacrum/src/lib.rs
21
68.48
-9.11% crates/iota-core/src/execution_cache.rs
18
88.71
1.16% crates/typed-store/src/database.rs
17
91.44
-1.07% crates/iota-json-rpc/src/governance_api.rs
16
89.04
crates/iota-core/src/authority/epoch_markers.rs
15
96.16
crates/iota-core/src/authority/historic_objects.rs
15
90.74
crates/iota-core/src/progress_logger.rs
15
95.89
crates/iota-core/src/rpc_indexes/schema.rs
11
86.21
0.22% crates/iota-json-rpc/src/read_api.rs
10
91.9
-1.48% crates/iota-core/src/checkpoints/checkpoint_executor/mod.rs
10
95.32
-0.51% crates/iota-core/src/par_index_live_object_set.rs
10
77.28
-1.6% crates/iota-core/src/storage.rs
9
95.43
crates/iota-core/src/authority/object_backlog_sweep.rs
9
0.0
0.0% crates/iota-indexer/src/apis/indexer_api.rs
9
60.53
0.84% crates/iota-json-rpc-types/src/iota_object.rs
9
85.57
85.57% crates/typed-store/src/memstore.rs
6
87.73
0.52% crates/iota-core/src/post_consensus_input_reader/reader.rs
6
51.73
-0.47% crates/iota-transactional-test-runner/src/simulator_persisted_store.rs
5
71.48
0.56% crates/iota/src/client_commands.rs
4
96.83
-3.17% crates/iota-core/src/transaction_outputs.rs
4
98.13
-1.49% crates/iota-types/src/effects/test_effects_builder.rs
4
78.34
-1.05% crates/test-cluster/src/lib.rs
3
92.43
0.73% crates/iota-core/src/execution_cache/writeback_cache.rs
3
92.11
crates/iota-core/src/index_rebuild_cancellation.rs
3
96.92
1.76% crates/iota-json-rpc/src/transaction_builder_api.rs
3
0.0
0.0% crates/iota-open-rpc/src/examples.rs
3
82.63
-0.03% crates/iota-swarm/src/memory/swarm.rs
3
74.54
-0.25% crates/iota-types/src/object.rs
2
85.29
0.07% crates/iota-core/src/global_state_hasher.rs
2
0.0
0.0% crates/iota-indexer/src/apis/coin_api.rs
2
69.76
-8.39% crates/iota-json-rpc/src/authority_state.rs
2
91.75
0.15% crates/iota-json-rpc/src/indexer_api.rs
2
86.56
17.81% crates/iota-sdk/src/apis/coin_read.rs
2
92.19
-2.55% crates/iota-snapshot/src/restore.rs
2
0.0
0.0% crates/iota-tool/src/commands.rs
2
71.8
-0.14% crates/iota-transaction-builder/src/lib.rs
2
96.42
-0.95% crates/typed-store/src/rocks/mod.rs
1
81.86
-0.4% crates/iota-snapshot/src/lib.rs

Coverage Regressions

Lines Coverage ∆ File
186
71.48
0.56% crates/iota/src/client_commands.rs
127
92.23
0.1% crates/iota-protocol-config/src/lib.rs
58
87.78
0.59% crates/iota-core/src/authority.rs
49
68.48
-9.11% crates/iota-core/src/execution_cache.rs
31
77.89
-10.88% crates/iota-types/src/effects/mod.rs
29
78.34
-1.05% crates/test-cluster/src/lib.rs
23
96.02
0.31% crates/starfish/core/src/commit_solidifier.rs
21
98.08
0.0% crates/starfish/core/src/commit_observer.rs
19
76.79
-4.85% crates/iota-json-rpc-types/src/iota_move.rs
15
78.47
6.85% crates/iota-core/src/authority/authority_store.rs
14
90.37
0.84% crates/iota-core/src/transaction_orchestrator.rs
13
95.86
-0.27% crates/starfish/core/src/authority_service.rs
11
77.28
-1.6% crates/iota-core/src/storage.rs
9
55.43
-5.14% crates/iota-json-rpc-types/src/iota_event.rs
9
94.37
-0.22% crates/starfish/core/src/header_synchronizer.rs
8
69.76
-8.39% crates/iota-json-rpc/src/authority_state.rs
8
76.68
-0.26% crates/iota-network/src/randomness/mod.rs
8
90.51
0.18% iota-execution/latest/iota-adapter/src/adapter.rs
7
92.43
0.73% crates/iota-core/src/execution_cache/writeback_cache.rs
7
97.06
-0.38% crates/iota-json-rpc/src/coin_api.rs
7
83.33
-8.97% crates/iota-types/src/inner_temporary_store.rs
6
94.94
-1.79% crates/iota-core/src/execution_cache/object_locks.rs
6
93.68
-2.63% crates/iota-core/src/transaction_driver/mod.rs
6
69.93
-3.92% crates/iota-json-rpc-types/src/lib.rs
4
95.87
-3.31% crates/iota-json-rpc/src/metrics.rs
4
89.87
-0.93% crates/iota-transaction-builder/src/utils.rs
4
88.71
1.16% crates/typed-store/src/database.rs
3
92.45
-0.11% crates/iota-core/src/authority/authority_per_epoch_store.rs
3
92.64
5.81% crates/iota-core/src/authority/authority_store_tables.rs
3
87.69
0.58% crates/iota-core/src/authority_server/validator_v2.rs
3
86.84
-0.88% crates/iota-types/src/base_types.rs
3
92.62
0.0% crates/iota-types/src/message_envelope.rs
2
85.32
0.77% crates/iota-core/src/checkpoints/mod.rs
2
96.77
-0.32% crates/iota-core/src/overload_monitor.rs
2
91.44
-1.07% crates/iota-json-rpc/src/governance_api.rs
2
44.69
3.2% crates/iota-tool/src/db_tool/db_dump.rs
2
35.05
35.05% crates/iota-tool/src/db_tool/index_search.rs
2
96.42
-0.95% crates/typed-store/src/rocks/mod.rs
2
87.5
-0.5% iota-execution/src/lib.rs
1
76.19
-0.95% crates/iota-data-ingestion-core/src/reader/fetch.rs
1
0.0
0.0% crates/iota-indexer/src/apis/indexer_api.rs
1
51.02
-2.04% crates/iota-json-rpc/src/logger.rs
1
60.53
0.84% crates/iota-json-rpc-types/src/iota_object.rs
1
89.04
-0.14% crates/iota-network/src/discovery/mod.rs
1
0.0
0.0% crates/iota-node/src/main.rs
1
94.67
-1.22% crates/iota-sdk/src/apis/quorum_driver.rs
1
92.19
-2.55% crates/iota-snapshot/src/restore.rs
1
73.81
-1.19% crates/iota-storage/src/sharded_lru.rs
1
74.06
0.94% crates/iota-types/src/error.rs
1
74.54
-0.25% crates/iota-types/src/object.rs
1
97.06
-0.98% crates/starfish/core/src/storage/mod.rs
1
72.0
-4.0% iota-execution/latest/iota-adapter/src/type_layout_resolver.rs
1
87.5
-0.74% iota-execution/src/verifier.rs
Jobs
ID Job ID Ran Files Coverage
1 nextest+simtest - 40983832694.1 1140
72.31
GitHub Action Run
Source Files on build 40983832694