kubeflow / sdk / 21944504251

73%

push

github

feat: add SparkClient API for SparkConnect session management (#225)

* feat(spark): add core types, dataclasses, and constants

- Add SparkConnectInfo, SparkConnectState, Driver, Executor types
- Add type tests for validation
- Add Kubernetes backend constants (CRD group, version, defaults)

Signed-off-by: Shekhar Rajak <shekharrajak@live.com>

* feat(spark): add backend base class and options pattern

- Add RuntimeBackend abstract base class with session lifecycle methods
- Add options pattern (Name, Image, Timeout, etc.) aligned with trainer SDK
- Add validation utilities for connect parameters
- Add comprehensive option tests

Signed-off-by: Shekhar Rajak <shekharrajak@live.com>

* feat(spark): add KubernetesBackend for SparkConnect CRD operations

- Implement KubernetesBackend with create/get/list/delete session methods
- Add port-forward support for out-of-cluster connections
- Add CRD builder utilities and URL validation
- Add comprehensive backend and utils tests with parametrized patterns

Signed-off-by: Shekhar Rajak <shekharrajak@live.com>

* feat(spark): add SparkClient API with KEP-107 compliant connect method

- Implement SparkClient as main user interface for SparkConnect sessions
- Support connect to existing server (base_url) or auto-create new session
- Add public exports for SparkClient, Driver, Executor, options
- Add SparkClient unit tests

Signed-off-by: Shekhar Rajak <shekharrajak@live.com>

* chore(spark): add test infrastructure and package init files

- Add test common utilities and fixtures
- Add package __init__ files for test directories
- Setup test/e2e/spark structure

Signed-off-by: Shekhar Rajak <shekharrajak@live.com>

* feat(spark): add example scripts demonstrating SparkClient usage

- Add spark_connect_simple.py with 3 usage levels (minimal, simple, advanced)
- Add spark_advanced_options.py with full configuration examples
- Add connect_existing_session.py for connecting to existing servers
- Add demo and test scri... (continued)

1115 of 1292 new or added lines in 18 files covered. (86.3%)

4092 of 5619 relevant lines covered (72.82%)

1.46 hits per line