3227
78%

Ran 04 Jun 2018 09:27PM UTC

Jobs 1

Files 101

Run time 7s

Badge

Embed ▾

pending completion

Build # 3227

Build Type

push

travis-ci

Committed by

ojarjur

Commit Message

Fix an MLToolbox preprocessing failure caused by pickling of TF objects. (#689)

It seems happening after we upgrade DataFlow version to 2.0. The issue happens when:

You have "import tensorflow as tf" in a DoFn's init.
At run time, DataFlow initializes the DoFn first, and then pickle and unpickle the instance to multiple workers. That means it needs to pickle the state including imports.
Under tensorflow there are some objects that are not pickle friendly. So pickling everything under tensorflow namespace causes it to fail at run time.
The issue also occur if you have global "import tensorflow" statement.

The fix is to move tensorflow import into either DoFn's process(), or start_bundle() function. That way the import occurs for each instance on each worker and no tensorflow pickling is needed.

Run Details

5444 of 6993 relevant lines covered (77.85%)

0.78 hits per line

Jobs

ID	Job ID	Ran	Files	Coverage
4	3227.4 (TOX_ENV=coveralls)	04 Jun 2018 09:27PM UTC	0	77.85	Travis Job 3227.4

Source Files on build 3227

Detailed source file information is not available for this build.

googledatalab / pydatalab / 3227
78%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 3227

googledatalab / pydatalab / 3227 78%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 3227

googledatalab / pydatalab / 3227
78%

README BADGES
x