iotaledger / iota / 22698035098
65%

DEFAULT BRANCH: develop
Ran
Jobs 1
Files 1540
Run time 3min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 63.852% (-0.2%) from 64.028%
22698035098

push

github

web-flow 
feat(iota-benchmark): end-to-end Abstract Account (AA) workload support in iota-aws-orchestrator + iota-benchmark (#9942)

This PR upgrades the AWS benchmark workflow to run Abstract Account (AA)
benchmarks end-to-end on an AWS test setup.

Previously, the AWS orchestrator flow mainly supported the legacy bench
workloads. With this PR, the orchestrator can also run the AA workload
and collect the same quality of reports and node-side latency spans
(Tempo/Grafana) that we use for other benchmarks.

Confluence (design / docs):
[iota-benchmark](https://iotafoundation.atlassian.net/wiki/spaces/LSC/pages/2876538895/iota-benchmark+Stress+Testing+Framework+Technical+Overview),
[aws-orchestrator](https://iotafoundation.atlassian.net/wiki/spaces/LSC/pages/2875752493/AWS+Orchestrator+iota-aws-orchestrator+-+Technical+Overview+DRAFT).
Confluence (benchmark reports): [local
report](https://iotafoundation.atlassian.net/wiki/spaces/LSC/pages/2746548229/Report+A+-+Local+benchmark+iota-benchmark+-+Node-side+metrics+comparison+of+Standard+Ed25519+transactions+vs+AA+Ed25519+via+MoveAuthenticator+in+IOTA#Abstract),
[aws
report](https://iotafoundation.atlassian.net/wiki/spaces/LSC/pages/2866577409/Report+B+-+AWS+benchmark+-+Node-side+metrics+comparison+of+Standard+Ed25519+transactions+vs+AA+Ed25519+via+MoveAuthenticator+in+IOTA+DRAFT).

# Description of change

### Orchestrator changes

- The iota-aws-orchestrator benchmark command now supports an AA mode.
- In AA mode, the orchestrator provisions the AWS cluster, boots
validators/fullnode, runs the AA load generator on the client, and
collects:
- Prometheus metrics and benchmark stats,
- Tempo/Grafana span latency distributions.

### AA benchmark configuration

- The benchmark CLI adds AA-specific flags (forwarded to the final
stress command), including:
- authenticator selection,
- owned vs shared payload mode,
- worker count and in-flight limits.

### High-level run flow

1. Provision AWS cluster (validators, fullnode... (continued)

56 of 1288 new or added lines in 18 files covered. (4.35%)

482 existing lines in 29 files now uncovered.

227190 of 355808 relevant lines covered (63.85%)

2762710.34 hits per line

New Missed Lines in Diff

Lines Coverage ∆ File
1
0.0
 0.0% crates/iota-aws-orchestrator/src/ssh.rs
1
0.0
 0.0% crates/iota-benchmark/src/workloads/expected_failure.rs
7
0.0
 0.0% crates/iota-benchmark/src/workloads/workload.rs
11
0.0
 0.0% crates/iota-benchmark/src/bin/stress.rs
26
0.0
 0.0% crates/iota-aws-orchestrator/src/main.rs
27
0.0
 0.0% crates/iota-benchmark/src/drivers/mod.rs
31
0.0
 0.0% crates/iota-benchmark/src/workloads/workload_configuration.rs
40
0.0
 0.0% crates/iota-benchmark/src/drivers/bench_driver.rs
46
0.0
 crates/iota-benchmark/src/options.rs
46
0.0
 crates/iota-benchmark/src/workloads/abstract_account/types.rs
58
59.29
 -3.99% crates/iota-aws-orchestrator/src/measurement.rs
68
63.1
 -7.53% crates/iota-aws-orchestrator/src/benchmark.rs
90
0.0
 0.0% crates/iota-aws-orchestrator/src/protocol/iota.rs
101
0.0
 crates/iota-benchmark/src/workloads/abstract_account/mod.rs
137
0.0
 0.0% crates/iota-aws-orchestrator/src/orchestrator.rs
147
0.0
 0.0% crates/iota-aws-orchestrator/src/monitor.rs
160
0.0
 crates/iota-benchmark/src/workloads/abstract_account/payload.rs
235
0.0
 crates/iota-benchmark/src/workloads/abstract_account/utils.rs

Uncovered Existing Lines

Lines Coverage ∆ File
1
85.0
 -0.45% consensus/core/src/subscriber.rs
1
59.29
 -3.99% crates/iota-aws-orchestrator/src/measurement.rs
1
0.0
 0.0% crates/iota-benchmark/src/drivers/bench_driver.rs
1
0.0
 0.0% crates/iota-benchmark/src/drivers/mod.rs
1
83.78
 0.0% crates/iota-cluster-test/src/test_case/native_transfer_test.rs
1
93.12
 -0.07% crates/starfish/core/src/transactions_synchronizer.rs
2
94.92
 1.78% crates/iota-core/src/consensus_adapter.rs
2
95.22
 -0.87% crates/iota-http/src/lib.rs
2
83.64
 -0.22% crates/starfish/core/src/commit_syncer/fast.rs
3
0.0
 0.0% crates/iota-aws-orchestrator/src/orchestrator.rs
3
77.05
 -4.92% crates/iota-core/src/traffic_controller/nodefw_test_server.rs
3
89.44
 -1.67% crates/iota-surfer/src/surfer_state.rs
3
70.14
 -0.6% crates/iota-swarm/src/memory/swarm.rs
4
79.65
 -0.46% crates/iota-core/src/authority_server.rs
4
85.57
 -1.37% crates/iota-core/src/authority/test_authority_builder.rs
4
83.64
 1.82% crates/iota-surfer/src/surf_strategy.rs
4
83.25
 -0.49% crates/iota-types/src/base_types.rs
5
0.0
 0.0% crates/iota-aws-orchestrator/src/protocol/iota.rs
5
65.77
 -1.22% crates/starfish/core/src/commit_syncer/mod.rs
6
76.3
 -1.3% crates/iota-core/src/storage.rs
6
72.22
 -3.7% crates/iota-json-rpc/src/lib.rs
7
88.6
 -0.97% crates/iota-core/src/authority/consensus_quarantine.rs
8
91.41
 -0.39% crates/starfish/core/src/header_synchronizer.rs
13
90.75
 -0.66% consensus/core/src/synchronizer.rs
13
76.24
 1.31% crates/test-cluster/src/lib.rs
38
78.87
 -1.43% crates/typed-store-derive/src/lib.rs
39
87.27
 1.47% consensus/core/src/authority_service.rs
65
93.12
 0.25% crates/starfish/core/src/authority_service.rs
237
87.07
 0.13% crates/iota-core/src/authority/authority_per_epoch_store.rs
Jobs
ID Job ID Ran Files Coverage
1 nextest+simtest - 22698035098.1 1552
63.6
 GitHub Action Run
Source Files on build 22698035098