great-expectations / great_expectations / 4945

78%

develop: 0%

Pull #1239

travis-ci

Add datasource -> dataset options and persist feature

* Add option "persist", true by default, for SparkDFDataset to persist the DataFrame it is passed. This addresses #1133
in a deeper way (thanks @tejsvirai for the robust debugging support and reproduction on spark).
  - Disabling this option should *only* be done if the user has *already* externally persisted the DataFrame, or if the
dataset is too large to persist but *computations are guaranteed to be stable across jobs*.
* Enable passing dataset kwargs through datasource via dataset_options batch_kwarg.

Signed-off-by: James Campbell <james.p.campbell@gmail.com>

20 of 20 new or added lines in 6 files covered. (100.0%)

9359 of 12030 relevant lines covered (77.8%)

2.33 hits per line