26043728131
52%

Ran 18 May 2026 04:06PM UTC

Jobs 2

Files 219

Run time 6min

Badge

Embed ▾

Committed 18 May 2026 03:37PM UTC coverage: 51.706% (-0.4%) from 52.076%

Build # 26043728131

Build Type

push

github

Committed by

hariharan-devarajan

Commit Message

feat(perf): performance improvements for parallel reading, indexing, and aggregation

Indexer
- Streaming parse-and-emit worker pipeline with bounded memory usage
- Concurrent SST artifact ingestion with staging support
- Gzip member slicing for parallel indexing
- Lazy decoding for compressed value counts
- Bypass DOM wrapper for indexer hot path (simdjson on_demand)
- Decoupled write workers from parse workers
- --rebuild-summaries flag and optimized root summary rebuild

Aggregator / MPI
- Task-based DAG execution for aggregator pipeline
- Shared staging for multi-node artifact relocation
- Per-node thread scaling to avoid oversubscription
- Unified distributed aggregation tracking, removed manifest consolidation
- Deterministic aggregation and intra-file parallelism

Trace reader / query
- Compiled predicate evaluation for AND-of-EQ queries
- Uniform-match shortcut for AND-of-EQ queries
- Line-range support for work items and checkpoint processing
- Optimized chunk pruning and checkpoint handling

Replay
- Pipelined replay with coroutines and channels
- JsonParser-based trace processing
- Optimized string handling and i/o buffering

Organize / writer / dft
- Parallel slice creation and merging in organize visitor
- Inline indexer in organize
- Gzip member tracking in writer
- Coroutine-based event dispatcher with extracted parse logic
- Batch flushing in organize visitor

Arrow / call_tree
- Optimized arrow conversion
- Arrow IPC support and improved save/load in call_tree

Build / infrastructure
- zlib-ng option, system simdjson fallback
- cgroup v1/v2 memory limit detection
- Auto-computed per-file memory estimates and batch sizes
- CI: perf branch trigger, formatting

Docs
- Rewritten indexer and trace reader API references

Coverage Stats

35907 of 90345 branches covered (39.74%)

Branch coverage included in aggregate %.

16869 of 21880 new or added lines in 137 files covered. (77.1%)

273 existing lines in 39 files now uncovered.

32021 of 41028 relevant lines covered (78.05%)

13164.29 hits per line

Uncovered Changes

Lines	Coverage	∆	File
708	35.77		src/dftracer/utils/python/batch_indexer.cpp
600	0.72		src/dftracer/utils/python/sst_distribution.cpp
356	51.2	-5.58%	src/dftracer/utils/utilities/indexer/index_database.cpp
343	46.7	9.14%	src/dftracer/utils/python/trace_reader.cpp
166	2.01		src/dftracer/utils/python/index_database.cpp
164	54.39		src/dftracer/utils/utilities/indexer/index_database_writer_context.cpp
163	51.55	-12.1%	src/dftracer/utils/utilities/reader/trace_reader.cpp
162	30.65	1.65%	src/dftracer/utils/binaries/dftracer_stats.cpp
156	38.03	-23.92%	src/dftracer/utils/utilities/composites/dft/aggregators/aggregator_utility.cpp
130	23.85	3.16%	src/dftracer/utils/python/json.cpp
113	41.81		src/dftracer/utils/python/schema_reconcile.cpp
111	51.72	-9.23%	src/dftracer/utils/utilities/composites/dft/indexing/chunk_pruner_utility.cpp
95	52.56	36.01%	src/dftracer/utils/utilities/composites/dft/aggregators/perfetto_trace_writer_utility.cpp
88	56.83		src/dftracer/utils/utilities/composites/dft/aggregators/aggregation_visitor.cpp
85	2.21		src/dftracer/utils/python/arrow_parallel_reader.cpp
83	17.13	-0.73%	src/dftracer/utils/utilities/composites/dft/comparator/comparison_config.cpp
78	57.48		src/dftracer/utils/utilities/composites/dft/aggregators/event_aggregator.cpp
67	21.93	-5.3%	src/dftracer/utils/utilities/composites/dft/aggregators/association_tracker.cpp
62	31.98		src/dftracer/utils/utilities/common/arrow/partition_router.cpp
62	48.7		src/dftracer/utils/utilities/indexer/index_database_sst_writer_context.cpp
53	70.9		src/dftracer/utils/utilities/composites/dft/visitors/bloom_visitor.cpp
53	37.47	3.8%	src/dftracer/utils/utilities/replay/replay.cpp
52	33.33	33.33%	src/dftracer/utils/utilities/call_tree/json_serializer.cpp
47	50.57		src/dftracer/utils/python/arrow_stream_capsule.cpp
47	42.62		src/dftracer/utils/utilities/common/arrow/partition_writer.cpp
46	44.54	30.65%	src/dftracer/utils/utilities/call_tree/call_tree_internal.cpp
45	0.0	0.0%	src/dftracer/utils/utilities/call_tree/call_tree.cpp
43	48.06	-8.57%	src/dftracer/utils/binaries/dftracer_organize.cpp
41	58.47	-2.15%	src/dftracer/utils/python/utilities/aggregator.cpp
38	49.04	-8.96%	src/dftracer/utils/utilities/common/arrow/ipc_writer.cpp
36	70.59		src/dftracer/utils/utilities/common/arrow/ipc_reader.cpp
31	62.24	10.31%	src/dftracer/utils/binaries/dftracer_replay.cpp
31	67.65	-3.56%	src/dftracer/utils/utilities/common/arrow/column_builder.cpp
31	0.0		src/dftracer/utils/utilities/composites/dft/reorganize/manifest_extractor.cpp
30	58.29		src/dftracer/utils/utilities/composites/dft/reorganize/group_writer_task.cpp
27	63.69		src/dftracer/utils/core/common/memory_budget.cpp
27	41.57		src/dftracer/utils/python/memoryview_batch.cpp
26	40.86		src/dftracer/utils/utilities/common/json/parser.cpp
23	51.22		src/dftracer/utils/python/streaming_iterator.cpp
22	76.89		src/dftracer/utils/utilities/composites/dft/aggregators/aggregation_serialization.cpp
21	41.76		src/dftracer/utils/utilities/composites/dft/reorganize/organize_visitor.cpp
20	51.5	-2.06%	src/dftracer/utils/binaries/dftracer_aggregator.cpp
19	47.69	47.69%	src/dftracer/utils/core/utils/timer.cpp
19	57.4		src/dftracer/utils/utilities/composites/dft/indexing/index_resolver_utility.cpp
19	61.05		src/dftracer/utils/utilities/composites/dft/reorganize/reconstructor_utility.cpp
19	61.39	0.75%	src/dftracer/utils/utilities/reader/internal/streams/line_stream.h
18	46.9		src/dftracer/utils/utilities/fileio/parallel/striped_writer.cpp
17	55.28	-16.56%	src/dftracer/utils/utilities/indexer/index_builder_utility.cpp
16	36.69	-33.63%	src/dftracer/utils/utilities/indexer/internal/gzip/gzip_indexer.cpp
15	46.34	10.63%	src/dftracer/utils/server/viz_api.cpp
14	53.83		src/dftracer/utils/binaries/dftracer_call_tree.cpp
14	61.2		src/dftracer/utils/utilities/call_tree/call_tree_save_arrow.cpp
14	57.44		src/dftracer/utils/utilities/call_tree/call_tree_save_binary.cpp
14	0.0	0.0%	src/dftracer/utils/utilities/composites/dft/statistics/chunk_detail_scanner_utility.cpp
14	65.14	-8.09%	src/dftracer/utils/utilities/composites/dft/views/view_definition.cpp
13	52.14	-1.77%	src/dftracer/utils/binaries/dftracer_info.cpp
12	43.93	-11.82%	src/dftracer/utils/binaries/dftracer_event_count.cpp
12	72.68	-2.68%	src/dftracer/utils/utilities/composites/dft/comparator/tree_table_formatter.cpp
11	63.3	0.87%	src/dftracer/utils/python/indexer.cpp
9	62.94	-2.13%	src/dftracer/utils/core/rocksdb/database.cpp
9	72.9		src/dftracer/utils/python/streaming_iterator.h
8	67.55	2.98%	src/dftracer/utils/python/trace_reader_iterator.cpp
8	36.0	-64.0%	src/dftracer/utils/python/trace_reader_iterator.h
8	82.69	9.38%	src/dftracer/utils/utilities/composites/dft/indexing/chunk_statistics.cpp
8	68.84		src/dftracer/utils/utilities/composites/dft/visitors/hash_table_visitor.cpp
7	60.6	1.7%	src/dftracer/utils/binaries/dftracer_comparator.cpp
7	51.24	-0.41%	src/dftracer/utils/python/runtime.cpp
7	75.06		src/dftracer/utils/utilities/composites/dft/aggregators/aggregation_logic.cpp
7	71.17		src/dftracer/utils/utilities/composites/dft/aggregators/system_metrics_serialization.cpp
7	78.06		src/dftracer/utils/utilities/composites/dft/visitors/manifest_visitor.cpp
7	53.09	-4.84%	src/dftracer/utils/utilities/indexer/provenance_database.cpp
6	74.27		src/dftracer/utils/binaries/common_cli.h
6	66.2		src/dftracer/utils/utilities/composites/dft/aggregators/aggregation_merge_operator.cpp
5	57.43	18.41%	src/dftracer/utils/utilities/composites/dft/views/view_reader_utility.cpp
5	72.36		src/dftracer/utils/utilities/fileio/parallel/layout.cpp
4	40.91		src/dftracer/utils/core/utils/timer.h
4	81.08	0.31%	src/dftracer/utils/python/arrow_helpers.cpp
4	55.2	-3.28%	src/dftracer/utils/python/utilities/comparator.cpp
4	0.0	0.0%	src/dftracer/utils/server/cursor.cpp
4	81.14		src/dftracer/utils/utilities/composites/dft/aggregators/aggregation_augmentation.cpp
4	78.77	0.47%	src/dftracer/utils/utilities/composites/dft/statistics/detailed_statistics.cpp
4	52.55		src/dftracer/utils/utilities/fileio/parallel/padded_striped_writer.cpp
3	89.26		src/dftracer/utils/utilities/common/statistics/timestamp_histogram.cpp
3	85.03	3.24%	src/dftracer/utils/utilities/composites/dft/indexing/chunk_dimension_stats.cpp
2	67.43	0.6%	src/dftracer/utils/core/pipeline/executor.cpp
2	63.19	5.9%	src/dftracer/utils/core/runtime.cpp
2	92.27	6.23%	src/dftracer/utils/utilities/composites/dft/aggregators/aggregation_metrics.cpp
2	76.39		src/dftracer/utils/utilities/composites/dft/aggregators/system_metrics_merge_operator.cpp
2	42.54	-20.53%	src/dftracer/utils/utilities/composites/dft/statistics/statistics_aggregator_utility.cpp
2	64.23		src/dftracer/utils/utilities/indexer/internal/index_batch_writer.h
1	60.62	3.52%	src/dftracer/utils/binaries/dftracer_gen_fake_trace.cpp
1	60.8	0.14%	src/dftracer/utils/core/rocksdb/filesystem.cpp
1	98.59	-1.41%	src/dftracer/utils/python/dftracer_utils_ext.cpp
1	18.39	0.12%	src/dftracer/utils/utilities/common/json/json_value.cpp
1	47.86	-0.79%	src/dftracer/utils/utilities/common/query/query.cpp
1	90.52	3.31%	src/dftracer/utils/utilities/composites/dft/indexing/bloom_filter.cpp
1	67.69	5.53%	src/dftracer/utils/utilities/composites/dft/indexing/chunk_indexer_utility.cpp
1	70.3	-2.07%	src/dftracer/utils/utilities/composites/dft/statistics/statistics_query_utility.cpp
1	54.9		src/dftracer/utils/utilities/fileio/parallel/sharded_writer.cpp

Coverage Regressions

Lines	Coverage	∆	File
68	30.65	1.65%	src/dftracer/utils/binaries/dftracer_stats.cpp
26	51.2	-5.58%	src/dftracer/utils/utilities/indexer/index_database.cpp
25	37.47	3.8%	src/dftracer/utils/utilities/replay/replay.cpp
17	46.7	9.14%	src/dftracer/utils/python/trace_reader.cpp
11	53.09	-4.84%	src/dftracer/utils/utilities/indexer/provenance_database.cpp
10	62.94	-2.13%	src/dftracer/utils/core/rocksdb/database.cpp
10	51.72	-9.23%	src/dftracer/utils/utilities/composites/dft/indexing/chunk_pruner_utility.cpp
9	44.54	30.65%	src/dftracer/utils/utilities/call_tree/call_tree_internal.cpp
8	60.6	1.7%	src/dftracer/utils/binaries/dftracer_comparator.cpp
8	0.0	-42.49%	src/dftracer/utils/utilities/composites/dft/aggregators/association_resolver_utility.cpp
6	65.5	-1.6%	src/dftracer/utils/core/pipeline/scheduler.cpp
6	0.0	0.0%	src/dftracer/utils/utilities/call_tree/call_tree.cpp
6	17.13	-0.73%	src/dftracer/utils/utilities/composites/dft/comparator/comparison_config.cpp
6	0.0	-67.15%	src/dftracer/utils/utilities/composites/dft/reorganize/event_router.cpp
6	55.28	-16.56%	src/dftracer/utils/utilities/indexer/index_builder_utility.cpp
5	67.65	-3.56%	src/dftracer/utils/utilities/common/arrow/column_builder.cpp
5	49.04	-8.96%	src/dftracer/utils/utilities/common/arrow/ipc_writer.cpp
5	61.39	0.75%	src/dftracer/utils/utilities/reader/internal/streams/line_stream.h
4	52.14	-1.77%	src/dftracer/utils/binaries/dftracer_info.cpp
4	72.68	-2.68%	src/dftracer/utils/utilities/composites/dft/comparator/tree_table_formatter.cpp
3	23.85	3.16%	src/dftracer/utils/python/json.cpp
3	58.47	-2.15%	src/dftracer/utils/python/utilities/aggregator.cpp
2	55.2	-3.28%	src/dftracer/utils/python/utilities/comparator.cpp
2	33.33	33.33%	src/dftracer/utils/utilities/call_tree/json_serializer.cpp
2	0.0	-12.5%	src/dftracer/utils/utilities/composites/dft/aggregators/aggregator_summary_utility.cpp
2	38.03	-23.92%	src/dftracer/utils/utilities/composites/dft/aggregators/aggregator_utility.cpp
2	90.52	3.31%	src/dftracer/utils/utilities/composites/dft/indexing/bloom_filter.cpp
1	43.93	-11.82%	src/dftracer/utils/binaries/dftracer_event_count.cpp
1	60.16	0.01%	src/dftracer/utils/binaries/dftracer_split.cpp
1	67.43	0.6%	src/dftracer/utils/core/pipeline/executor.cpp
1	60.8	0.14%	src/dftracer/utils/core/rocksdb/filesystem.cpp
1	37.46	-0.49%	src/dftracer/utils/core/tasks/task.cpp
1	51.93	-2.76%	src/dftracer/utils/core/tasks/task_result.cpp
1	63.3	0.87%	src/dftracer/utils/python/indexer.cpp
1	51.24	-0.41%	src/dftracer/utils/python/runtime.cpp
1	46.34	10.63%	src/dftracer/utils/server/viz_api.cpp
1	18.39	0.12%	src/dftracer/utils/utilities/common/json/json_value.cpp
1	0.0	0.0%	src/dftracer/utils/utilities/composites/dft/statistics/chunk_detail_scanner_utility.cpp
1	53.06	-4.94%	src/dftracer/utils/utilities/reader/internal/reader_factory.cpp

Jobs

ID	Job ID	Ran	Files	Coverage
1	ubuntu-22.04 - 26043728131.1	18 May 2026 04:17PM UTC	216	49.43	GitHub Action Run
2	macos-latest - 26043728131.2	18 May 2026 04:06PM UTC	214	50.39	GitHub Action Run

llnl / dftracer-utils / 26043728131
52%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Uncovered Changes

Coverage Regressions

Jobs

Source Files on build 26043728131

llnl / dftracer-utils / 26043728131 52%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Uncovered Changes

Coverage Regressions

Jobs

Source Files on build 26043728131

llnl / dftracer-utils / 26043728131
52%

README BADGES
x