zopefoundation / ZODB / 788

84%

Pull #183

travis-ci

Provide a way to retrieve raw extensions data

Currently when client has IStorageTransactionMetaData instance (e.g. on
storage iteration) it is possible to read transaction's .user and
.description in raw form, but .extension is always returned unpickled.

This creates several problems:

- tools like `zodb dump` [1] cannot dump data exactly as stored on a
  storage. This makes database potentially not bit-to-bit identical to
  its original after restoring from such dump.

- `zodb dump` output could be changing from run to run on the same
  database. This comes from the fact that e.g. python dictionaries are
  unordered and so when pickling a dict back to bytes the result could
  be not the same as original.

  ( this problem can be worked-around partly to work reliably for e.g.
    dict with str keys - by always emitting items in key sorted order,
    but it is hard to make it work reliably for arbitrary types )

Both issues make it hard to verify integrity of database at the lowest
possible level after restoration, and make it hard to verify bit-to-bit
compatibility with non-python ZODB implementations.

To fix we provide a way to retrieve raw extension from transaction
metadata. This is done in backward-compatible way by introducing to
IStorageTransactionMetaData new optional attribute

	.extension_bytes

which can be optionally provided by an IStorageTransactionMetaData to
indicate that transaction metadata can be also read in raw from.

Then ZODB.Connection.TransactionMetaData (the only in-tree
IStorageTransactionMetaData implementer) ctor is extended to see whether
it was given extension in raw form, and if yes, remember it as
.extension_bytes in addition to deserialized .extension dict.

This way storages that now pass raw extension to TransactionMetaData will
automatically provide .extension_bytes in their iterated transactions.

Teach FileStorage & friends to provide/use new functionality.

/helped-by: Jim Fulton.

[1] https://lab.nexedi.com/nexedi/zodbtools

9 of 9 new or added lines in 3 files covered. (100.0%)

6431 of 8531 relevant lines covered (75.38%)

3.0 hits per line