vocdoni / vocdoni-node / 8345233467

63%

push

github

vochain/indexer: move away from protojson

We are starting to use sqlite backup snapshots from the indexer
as a way to bootstrap one node from another, and this system requires
the snapshots to be deterministic so that they may be hashed.

We encountered issues where protojson-encoded column values
had inconsistent whitespace depending on each node,
and after some investigation, we found the culprit:
https://go-review.googlesource.com/c/protobuf/+/194037

Since protojson is explicitly non-deterministic, to the point that it
inserts random spaces depending on the hash of the binary,
avoid it entirely and use proto encoding instead.

We had chosen protojson for the same reason that we encode other
structured bits of data in encoding/json: sqlite has some native
support for JSON, meaning it is a bit more flexible than binary formats.

Even though proto is also not a stable format per se,
we already use it as a deterministic encoding in the state messages,
and it does follow a relatively strict specification,
so it seems to be deterministic enough for us in practice.

Note that this is a breaking change for indexer databases;
any node with an existing indexer database should throw it away.
The alternative was to write a SQL migration with Go code
to decode protojson and re-encode as proto, but we decided that wasn't
worth the effort.

19 of 19 new or added lines in 4 files covered. (100.0%)

15817 of 25955 relevant lines covered (60.94%)

38065.92 hits per line