getdozer / dozer
76%
main: 78%

LAST BUILD BRANCH: refs/tags/v0.1.39
DEFAULT BRANCH: main
Repo Added
Files 431
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH gh-readonly-queue/main/pr-1912-058f1ae3d5c2d578206794449a26faf614125a67
branch: gh-readonly-queue/main/pr-1912-058f1ae3d5c2d578206794449a26faf614125a67
CHANGE BRANCH
x

coverage: 76.247% (+0.8%) from 75.446%
5972853941

push

github

web-flow 
feat: make probabilistic optimizations optional and tunable in the YAML config (#1912)

Probabilistic optimization sacrifices accuracy in order to reduce memory consumption. In certain parts of the pipeline, a Bloom Filter is used ([set_processor](https://github.com/getdozer/dozer/blob/<a class=hub.com/getdozer/dozer/commit/<a class="double-link" href="https://git"><a class=hub.com/getdozer/dozer/commit/2e3ba96c3f4bdf9a691747191ab15617564d8ca2">2e3ba96c3/dozer-sql/src/pipeline/product/set/set_processor.rs#L20)), while in other parts, hash tables that store only the hash of the keys instead of the full keys are used ([aggregation_processor](https://github.com/getdozer/dozer/blob/2e3ba96c3f4bdf9a691747191ab15617564d8ca2/dozer-sql/src/pipeline/aggregation/processor.rs#L59) and [join_processor](https://github.com/getdozer/dozer/blob/2e3ba96c3f4bdf9a691747191ab15617564d8ca2/dozer-sql/src/pipeline/product/join/operator.rs#L57-L58)).

This commit makes these optimizations disabled by default and offers user-configurable flags to enable each of these optimizations separately.

This is an example of how to turn on probabilistic optimizations for each processor in the Dozer configuration.

```
flags:
  enable_probabilistic_optimizations:
    in_sets: true # enable probabilistic optimizations in set operations (UNION, EXCEPT, INTERSECT); Default: false
    in_joins: true # enable probabilistic optimizations in JOIN operations; Default: false
    in_aggregations: true # enable probabilistic optimizations in aggregations (SUM, COUNT, MIN, etc.); Default: false
```

347 of 347 new or added lines in 25 files covered. (100.0%)

47165 of 61858 relevant lines covered (76.25%)

48442.96 hits per line

Relevant lines Covered
61858 RELEVANT LINES 47165 COVERED LINES
48442.96 HITS PER LINE
Source Files on gh-readonly-queue/main/pr-1912-058f1ae3d5c2d578206794449a26faf614125a67

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
5972853941 gh-readonly-queue/main/pr-1912-058f1ae3d5c2d578206794449a26faf614125a67 feat: make probabilistic optimizations optional and tunable in the YAML config (#1912) Probabilistic optimization sacrifices accuracy in order to reduce memory consumption. In certain parts of the pipeline, a Bloom Filter is used ([set_processor]... push web-flow github
76.25
See All Builds (2270)