26043728131

Committed 18 May 2026 03:37PM UTC coverage: 51.706% (-0.4%) from 52.076%

Build # 26043728131

Build Type

push

github

Committed by

hariharan-devarajan

Commit Message

feat(perf): performance improvements for parallel reading, indexing, and aggregation

Indexer
- Streaming parse-and-emit worker pipeline with bounded memory usage
- Concurrent SST artifact ingestion with staging support
- Gzip member slicing for parallel indexing
- Lazy decoding for compressed value counts
- Bypass DOM wrapper for indexer hot path (simdjson on_demand)
- Decoupled write workers from parse workers
- --rebuild-summaries flag and optimized root summary rebuild

Aggregator / MPI
- Task-based DAG execution for aggregator pipeline
- Shared staging for multi-node artifact relocation
- Per-node thread scaling to avoid oversubscription
- Unified distributed aggregation tracking, removed manifest consolidation
- Deterministic aggregation and intra-file parallelism

Trace reader / query
- Compiled predicate evaluation for AND-of-EQ queries
- Uniform-match shortcut for AND-of-EQ queries
- Line-range support for work items and checkpoint processing
- Optimized chunk pruning and checkpoint handling

Replay
- Pipelined replay with coroutines and channels
- JsonParser-based trace processing
- Optimized string handling and i/o buffering

Organize / writer / dft
- Parallel slice creation and merging in organize visitor
- Inline indexer in organize
- Gzip member tracking in writer
- Coroutine-based event dispatcher with extracted parse logic
- Batch flushing in organize visitor

Arrow / call_tree
- Optimized arrow conversion
- Arrow IPC support and improved save/load in call_tree

Build / infrastructure
- zlib-ng option, system simdjson fallback
- cgroup v1/v2 memory limit detection
- Auto-computed per-file memory estimates and batch sizes
- CI: perf branch trigger, formatting

Docs
- Rewritten indexer and trace reader API references

Coverage Stats

35907 of 90345 branches covered (39.74%)

Branch coverage included in aggregate %.

16869 of 21880 new or added lines in 137 files covered. (77.1%)

273 existing lines in 39 files now uncovered.

32021 of 41028 relevant lines covered (78.05%)

13164.29 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

53.06

/src/dftracer/utils/utilities/reader/internal/reader_factory.cpp

#include <dftracer/utils/core/common/format_detector.h>
#include <dftracer/utils/core/common/logging.h>
#include <dftracer/utils/utilities/indexer/internal/gzip/gzip_indexer.h>
#include <dftracer/utils/utilities/indexer/internal/tar/tar_indexer.h>
#include <dftracer/utils/utilities/reader/internal/gzip_reader.h>
#include <dftracer/utils/utilities/reader/internal/reader_factory.h>
#include <dftracer/utils/utilities/reader/internal/tar_reader.h>

#include <stdexcept>

namespace dftracer::utils::utilities::reader::internal {

std::shared_ptr<Reader> ReaderFactory::create(const std::string &archive_path,
                                              const std::string &index_path,
                                              std::size_t index_ckpt_size) {
    ArchiveFormat format = FormatDetector::detect(archive_path);

    DFTRACER_UTILS_LOG_DEBUG(
        "ReaderFactory::create_reader - detected format: %d for file: %s",
        static_cast<int>(format), archive_path.c_str());

    switch (format) {
        case ArchiveFormat::GZIP:
            return std::make_shared<GzipReader>(archive_path, index_path,
                                                index_ckpt_size);

        case ArchiveFormat::TAR_GZ:
            return std::make_shared<TarReader>(archive_path, index_path,
                                               index_ckpt_size);

        default:
            throw std::runtime_error("Unsupported archive format for file: " +
                                     archive_path);
    }
}

std::shared_ptr<Reader> ReaderFactory::create(
    std::shared_ptr<dftracer::utils::utilities::indexer::internal::Indexer>
        indexer) {
    if (!indexer) {
        throw std::invalid_argument("Indexer cannot be null");
    }

    if (indexer->get_format_type() == ArchiveFormat::TAR_GZ) {
        return std::make_shared<TarReader>(
            std::static_pointer_cast<
                dftracer::utils::utilities::indexer::internal::tar::TarIndexer>(
                indexer));
    }

    return std::make_shared<GzipReader>(indexer);
}

bool ReaderFactory::is_format_supported(ArchiveFormat format) {
    switch (format) {
        case ArchiveFormat::GZIP:
        case ArchiveFormat::TAR_GZ:
            return true;
        default:
            return false;
    }
}
}  // namespace dftracer::utils::utilities::reader::internal

1	#include <dftracer/utils/core/common/format_detector.h>
2	#include <dftracer/utils/core/common/logging.h>
3	#include <dftracer/utils/utilities/indexer/internal/gzip/gzip_indexer.h>
4	#include <dftracer/utils/utilities/indexer/internal/tar/tar_indexer.h>
5	#include <dftracer/utils/utilities/reader/internal/gzip_reader.h>
6	#include <dftracer/utils/utilities/reader/internal/reader_factory.h>
7	#include <dftracer/utils/utilities/reader/internal/tar_reader.h>
8
9	#include <stdexcept>
10
11	namespace dftracer::utils::utilities::reader::internal {
12
13	std::shared_ptr<Reader> ReaderFactory::create(const std::string &archive_path,	958✔
14	const std::string &index_path,
15	std::size_t index_ckpt_size) {
16	ArchiveFormat format = FormatDetector::detect(archive_path);	958✔
17
18	DFTRACER_UTILS_LOG_DEBUG(
19	"ReaderFactory::create_reader - detected format: %d for file: %s",
20	static_cast<int>(format), archive_path.c_str());
21
22	switch (format) {	958!
23	case ArchiveFormat::GZIP:	478✔
24	return std::make_shared<GzipReader>(archive_path, index_path,	1,427✔
25	index_ckpt_size);	468✔
26
27	case ArchiveFormat::TAR_GZ:	1✔
28	return std::make_shared<TarReader>(archive_path, index_path,	3✔
29	index_ckpt_size);	1✔
30
31	default:
32	throw std::runtime_error("Unsupported archive format for file: " +	×
UNCOV 33	archive_path);	×
34	}
35	}	479✔
36
37	std::shared_ptr<Reader> ReaderFactory::create(	277✔
38	std::shared_ptr<dftracer::utils::utilities::indexer::internal::Indexer>
39	indexer) {
40	if (!indexer) {	277✔
41	throw std::invalid_argument("Indexer cannot be null");	×
42	}
43
44	if (indexer->get_format_type() == ArchiveFormat::TAR_GZ) {	278✔
45	return std::make_shared<TarReader>(	33!
46	std::static_pointer_cast<	11✔
47	dftracer::utils::utilities::indexer::internal::tar::TarIndexer>(	22✔
48	indexer));	11✔
49	}
50
51	return std::make_shared<GzipReader>(indexer);	256✔
52	}	139✔
53
54	bool ReaderFactory::is_format_supported(ArchiveFormat format) {	×
55	switch (format) {	×
56	case ArchiveFormat::GZIP:
57	case ArchiveFormat::TAR_GZ:
58	return true;	×
59	default:
60	return false;	×
61	}
62	}
63	} // namespace dftracer::utils::utilities::reader::internal

llnl / dftracer-utils / 26043728131

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous