wegamekinglc / calc-flow / 35460312235
93%

DEFAULT BRANCH: main
Ran
Jobs 2
Files 51
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 94.68% (+1.5%) from 93.179%
35460312235

push

github

web-flow 
feat: protobuf and custom decoders for Kafka payloads (#296)

* feat: decode protobuf Kafka payloads through runtime descriptors

The Kafka source gains a third payload format: one protobuf message per
record value, decoded through a FileDescriptorSet loaded once when the
source opens. The data-only options descriptor_set and message name the
serialized pool and the fully-qualified message; the explicit schema
projects scalar message fields by name onto the Arrow vocabulary, with
enums decoding to their value names. Proto3 implicit-presence fields
decode their defaults, explicit-presence fields decode null when
nullable and fail closed otherwise, and repeated, map, bytes, and
nested message fields fail closed with the field name preserved. The
codec registers as protobuf/1 alongside json and csv; sinks reject the
source-only format at option validation.

* feat: register custom Kafka payload decoders from Rust and Python

Custom Kafka sources select a trusted decoder through the data-only
decoder option identity and the custom/1 binding format, registered
globally by register_format_codecs. Rust transports implement
FormatDecoder and share a KafkaDecoderRegistry into the source factory
or register_kafka_connectors_with_decoders; registrations stay visible
through the frozen connector snapshot, so decoders may register until
the job opens. Option validation stays data-only: an unregistered
identity fails the source at open, built-in format names are reserved,
and descriptor_set/message/decoder companions are rejected outside
their own formats. Python applications register a callable taking the
payload bytes and returning a pyarrow RecordBatch or Table through
Runtime.register_kafka_decoder; the PyO3 wrapper enforces the explicit
schema and decode bounds and retains the callback root for GC.

* docs: add Kafka protobuf and custom-decoder examples

Add runnable coverage for both Kafka payload formats: the Python
27_kafka_protobuf_source.py and 28_kafka_c... (continued)

9877 of 10432 relevant lines covered (94.68%)

0.95 hits per line

Coverage Regressions

Lines Coverage ∆ File
30
91.26
-0.58% .venv/lib/python3.13/site-packages/calc_flow/pipeline.py
Jobs
ID Job ID Ran Files Coverage
1 studio - 35460312235.1 10
92.17
GitHub Action Run
2 python - 35460312235.2 41
95.32
GitHub Action Run
Source Files on build 35460312235