juarezr / spark-streaming-google-pubsub / 68

79%

Build Type

push

github

Committed by web-flow

Commit Message

Fix: tune auto receive time to achieve the maximum throughput possible with the available heap and RSS memory in the driver (#23)

- Make the connector discover and monitor the available memory limits
for the driver in the instance
- Adapt the message rate to the available memory to avoid out-of-memory
errors and being killed by the OOM killer
- Refine cycle time estimation logic to improve the throughput
- Add a new ackMode=AfterCheckpoint option to improve memory usage and
throughput
- Add Spark v4.3 support and testing in CI

Coverage Stats

1226 of 1711 branches covered (71.65%)

Branch coverage included in aggregate %.

862 of 1033 new or added lines in 11 files covered. (83.45%)

4 existing lines in 1 file now uncovered.

2315 of 2760 relevant lines covered (83.88%)

0.84 hits per line