4
78%
master: 78%

Ran 04 Jun 2018 09:27PM UTC

Files 101

Run time 6s

Badge

Embed ▾

Committed 04 Jun 2018 09:09PM UTC coverage: 77.849%. Remained the same

Job # TOX_ENV=coveralls

Build Type

push

travis-ci

Committed by

ojarjur

Commit Message

Fix an MLToolbox preprocessing failure caused by pickling of TF objects. (#689)

It seems happening after we upgrade DataFlow version to 2.0. The issue happens when:

You have "import tensorflow as tf" in a DoFn's init.
At run time, DataFlow initializes the DoFn first, and then pickle and unpickle the instance to multiple workers. That means it needs to pickle the state including imports.
Under tensorflow there are some objects that are not pickle friendly. So pickling everything under tensorflow namespace causes it to fail at run time.
The issue also occur if you have global "import tensorflow" statement.

The fix is to move tensorflow import into either DoFn's process(), or start_bundle() function. That way the import occurs for each instance on each worker and no tensorflow pickling is needed.

Run Details

5444 of 6993 relevant lines covered (77.85%)

0.78 hits per line

googledatalab / pydatalab / 3227 / 4
78%
master: 78%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 3227.4 (TOX_ENV=coveralls)

googledatalab / pydatalab / 3227 / 4 78% master: 78%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 3227.4 (TOX_ENV=coveralls)

googledatalab / pydatalab / 3227 / 4
78%
master: 78%

README BADGES
x