1
52%
master: 52%

Ran 07 Oct 2024 11:15AM UTC

Files 160

Run time 4s

Badge

Embed ▾

Committed 04 Oct 2024 02:17PM UTC coverage: 52.005% (+0.08%) from 51.928%

Job # 11214130791.1

Build Type

push

github

Committed by

spenes

Commit Message

Fix NPEs in TypesAccumulator

We've seen exceptions in spark executors like:

```
java.lang.NullPointerException: Cannot invoke "scala.collection.mutable.Set.isEmpty()" because the return value of "com.snowplowanalytics.snowplow.rdbloader.transformer.batch.spark.TypesAccumulator.accum()" is null
```

The error is coming from our Spark Accumulator for accumulating Iglu
types. This is similar to [an issue previously seen][1] in Spark's own
`CollectionAccumulator`. That issue [was fixed in Spark][2] by making
the accumulator's internal state non-final, and synchronizing access to
the internal state. So here we make the exact same change to our own
Accumulator.

It is a rare race condition which is hard to reproduce.

[1]: https://issues.apache.org/jira/browse/SPARK-20977
[2]: https://github.com/apache/spark/pull/31540

Run Details

2321 of 4463 relevant lines covered (52.01%)

0.83 hits per line

snowplow / snowplow-rdb-loader / 11214130791 / 1
52%
master: 52%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 11214130791.1

snowplow / snowplow-rdb-loader / 11214130791 / 1 52% master: 52%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 11214130791.1

snowplow / snowplow-rdb-loader / 11214130791 / 1
52%
master: 52%

README BADGES
x