snowplow / snowplow-rdb-loader / 11214130791

52%

master: 52%

Build Type

push

github

Committed by spenes

Commit Message

Fix NPEs in TypesAccumulator

We've seen exceptions in spark executors like:

```
java.lang.NullPointerException: Cannot invoke "scala.collection.mutable.Set.isEmpty()" because the return value of "com.snowplowanalytics.snowplow.rdbloader.transformer.batch.spark.TypesAccumulator.accum()" is null
```

The error is coming from our Spark Accumulator for accumulating Iglu
types. This is similar to [an issue previously seen][1] in Spark's own
`CollectionAccumulator`. That issue [was fixed in Spark][2] by making
the accumulator's internal state non-final, and synchronizing access to
the internal state. So here we make the exact same change to our own
Accumulator.

It is a rare race condition which is hard to reproduce.

[1]: https://issues.apache.org/jira/browse/SPARK-20977
[2]: https://github.com/apache/spark/pull/31540

Run Details

13 of 14 new or added lines in 1 file covered. (92.86%)

11 existing lines in 9 files now uncovered.

2321 of 4463 relevant lines covered (52.01%)

0.83 hits per line