googledatalab / pydatalab / 2095
78%

DEFAULT BRANCH: master
Ran
Jobs 1
Files 87
Run time 11min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
2095

push

travis-ci

web-flow 
Introducing support for the bq.load and bq.execute task-types in %pipeline (#488)

* Introducing Datalab Notebook Productionization with %pipeline (#480)

* Airflow python spec generation.

* Airflow python spec generation

* Deleting extraneous files from tests

* Removing TODO to pick either Yaml or Json. We'll allow both. No harm in that.

* Adding ability to specify 'query' instead of 'bql' for BigQuery.

* airflow_tests were being run twice in error. Fixing.

* Auto merged files after git rebase

* Resolving test conflicts.

* Fixing up test related changes

* Making new top-level schedule section in YAML spec; AirflowDag->AirflowPipeline rename

* dag_id -> pipeline_id

* Introducing BigQueryTableDeleteOperator, BigQueryToBigQueryOperator, and BigQueryToCloudStorageOperator

* _airflow_dag.py -> _airflow_pipeline.py

* Introducing %pipeline magic with support for "create" command. Also, some refactoring: AirflowPipeline -> Pipeline. Pipeline -> Deploy.

* Deleting airflow_job.py

* Importing the %%pipeline magic to kernels so that it's callable from datalab.

* Plugging into datalab object model to parse "pipeline create" magic.

* Stub implementation for pipeline load functionality

* BQ queries can now be referenced by name in the yaml spec!

* Some refactoring / clean-up.

* airflow -> pipeline

* Using datalab's parse_config for cell_body parsing (and throwing out _coders.py)

* Airflow spec now does not specify default params if the values are defaults.

* Removing "deploy" subcommand. This will be folded into create and there will be a single subcommand under the %pipeline magic.

* Some cleanup ahead of PR.

* Minor cleanup

* Adding doc strings, and fixing tests because python 3 renamed dict.iteritems -> dict.items

* More doc-strings

* Sorting dictionaries after iteration so that the resulting order is stable.

* Fixing lint issues

* Fixing flake8 errors (still a few to go)... (continued)

4520 of 5835 relevant lines covered (77.46%)

0.77 hits per line

Jobs
ID Job ID Ran Files Coverage
4 2095.4 (TOX_ENV=coveralls) 0
77.46
 Travis Job 2095.4
Source Files on build 2095
Detailed source file information is not available for this build.