zodb / relstorage / 1537

##############################################################################

#

# Copyright (c) 2008 Zope Foundation and Contributors.

# All Rights Reserved.

#

# This software is subject to the provisions of the Zope Public License,

# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.

# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED

# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS

# FOR A PARTICULAR PURPOSE.

#

##############################################################################

Stores pickles in the database.

"""

    'RelStorage',

]

        "does nothing"

                 ConflictResolution.ConflictResolvingStorage):

    """

    Storage to a relational database, based on invalidation polling.

    """

    # pylint:disable=too-many-instance-attributes,too-many-public-methods

    # ZODB TID of the last transaction committed by this instance.

    # _closed is True after self.close() is called.

    # _cache if set is a StorageCache object.

    # If the blob directory is set, blobhelper is a BlobHelper.

    # The state of committing that were in. Certain operations are

    # only available in certain states; certain information is only needed

    # in certain states.

    # _instances is a list of weak references to storage instances bound

    # to the same database.

                 options=None, cache=None, blobhelper=None,

                 store_connection_pool=None,

                 **kwoptions):

        # pylint:disable=too-many-branches, too-many-statements

                            "an options parameter or keyword arguments, not both")

            # At the top level, not new_instance(), and not asked to create.

        # A ZODB Connection is documented as not being thread-safe and

        # must be used only by a single thread at a time. In IMVCC,

        # each Connection has a unique storage object. So the storage

        # object is used only by a single thread. So there's really

        # not much point in RelStorage keeping thread locks around its

        # methods.

        # There are a small handful of ZODB tests that assume the

        # storage is thread safe (but they know nothing about IMVCC);

        # for those tests we use a custom wrapper storage that makes

        # a single instance appear thread safe.

        else:

                # Use the database name as the cache prefix.

        else:

        # Now copy in a bunch of methods from our component objects.

        # Many of these are 'stale_aware', meaning that we can ask

        # them for a version of themselves that does something

        # different when we go stale. When that happens, we *replace*

        # the object in our __dict__ with the new one; and then

        # reverse that when we're no longer stale.

        #

        # The storage wrapper zc.zlibstorage also copies methods into

        # itself when it is created, from the storage it is wrapping.

        # Because of this, stale aware methods like history() do not

        # do the right thing when we're wrapped by zc.zlibstorage.

        else:

    def __repr__(self):

        return "<%s at %x keep_history=%s phase=%r blobhelper=%r cache=%r>" % (

            self.__class__.__name__,

            id(self),

            self.keep_history,

            self._tpc_phase,

            self.blobhelper,

            self._cache

        )

        """Creates and returns another storage instance.

        See ZODB.interfaces.IMVCCStorage.

        """

                           create=False, options=options, cache=cache,

                           blobhelper=blobhelper,

                           store_connection_pool=self._store_connection_pool.new_instance())

        # NOTE: We're depending on the GIL (or list implementation)

        # for thread safety here.

            # registerDB has been called on us but isn't called on

            # our children. Make sure any wrapper that needs to transform

            # records can do so.

            # See https://github.com/zodb/relstorage/issues/71

        """

        Return a historical connection for the given *before* tid.

        The connection is read-only, and that is enforced at this level.

        """

        # XXX This might need some work to better adapt the cache? We

        # can *know* that we're pinned to see older transactions, so

        # our TID doesn't need to move forward, and we don't need to

        # poll. We're read-only so we'll never be joined to a

        # transaction or commit, we'll never have invalidations. The

        # HistoricalStorageAdapter blocks sync() and

        # poll_invalidations() from being called on this instance, but

        # our load connection is going to do that automatically the

        # first time it's accessed after a transaction

        # rollback...which, since we're not joined to a transaction,

        # should never happen...except for afterCompletion(), which

        # the ZODB connection has called on it for every transaction

        # whether it's joined or not...but the HistoricalStorageAdapter

        # doesn't implement that *either*, so it never makes it down to

        # this level. This means our load connection can stay open and

        # un-rolled-back for quite a long time, which is probably not great.

        # In case we haven't polled yet.

        """Clear all objects and transactions out of the database.

        Used by the test suite and the ZODBConvert script.

        """

        """

        Release external resources used by this storage instance.

        This includes the database sessions (connections) and any memcache

        connections.

        Does *not* affect any other instances created by this instance. This object

        should still be :meth:`close` (but note that might have global affects

        on other instances of the same base object).

        """

        """Close the storage and all instances."""

        """Return a string that can be used to sort storage instances.

        The key must uniquely identify a storage and must be the same

        across multiple instantiations of the same storage.

        """

        """Return database size in bytes"""

                # Prior to ZODB 5, this would be called by the database itself.

                # (I wish it would still do that.)

                and not ZODB.interfaces.IDatabase.providedBy(wrapper)

                and not hasattr(type(wrapper), 'new_instance')):

            # Fixes for https://github.com/zopefoundation/zc.zlibstorage/issues/2

            # We special-case zlibstorage for speed

                # zc.zlibstorage has a custom copyTransactionsFrom that hides

                # our own implementation. It just uses ZODb.blob.copyTransactionsFromTo.

                # Use our implementation.

            else:

        # Prior to ZODB 4.3.1, ConflictResolvingStorage would raise an AttributeError

    # For the time between tpc_begin and tpc_abort, if anything we do

    # would trigger an exception to propagate up (store(), tpc_vote(),

    # whatever) we should immediately abort the transaction at the

    # database level and release locks. This will allow other

    # transactions to make progress faster. It also can help in the ZODB test suite,

    # since some places manually call the tpc_* methods and aren't careful to

    # tpc_abort() in the event of an exception like `transaction` is.

    #

    # Some of this is manual (in the tpc_* methods), but most of it is

    # done through the @phase_dependent_aborts_early decorator.

            # Could be a database (connection) error, could be a programming

            # bug. Either way, we're fine to roll everything back and hope

            # for the best on a retry. Perhaps we need to raise a TransientError?

            # tid is a committed transaction we will restore.

            # The allowed actions are carefully prescribed.

            # This argument is specified by IStorageRestoreable

            else:

        # Returns an iterable of OIDs; the storage will ghost all

        # cached objects in that list. This is invalidation because

        # the object has changed during the commit process, due to

        # conflict resolution or undo.

        else:

            # OH NO! This isn't supposed to happen!

            # It's unlikely tpc_abort will get called...

        # The store connection is either committed or rolledback;

        # the load connection is now rolledback.

        # It might be nice to de-dup this so we're not storing it both

        # in the phase and in self, but if it was needed during TPC,

        # when our phase is not ``ITPCStateNotInTransaction``, we couldn't

        # get it.

        # _force is not a public argument, it is an internal

        # implementation detail.

                # We're here under unexpected circumstances. It's possible something

                # might go wrong rolling back.

            # We haven't committed *or* polled for transactions,

            # so our MVCC state is "floating".

            # Read directly from the database to get the latest value,

                   int64_to_8bytes(self._cache.highest_visible_tid or 0))

        # This is called from ``Connection.add`` which can be called at any time

        # by the application, so we don't know what state we're in. It is also called from

        # ``Connection._commit``, which is called during TPC.

        # If we're committing, we can't restart the connection.

        # XXX: This is broken for purposes of copyTransactionsFrom() because

        # it can only be iterated over once. zodbconvert works around this.

            self._adapter, self._load_connection, start, stop)

        """

        Implementation of `ZODB.interfaces.IStorageCurrentRecordIteration`.

        .. caution::

           You must completely consume the iteration.

        """

        # The interface doesn't define any semantics for *next*; it's

        # completely up to the storage and is just a magic cookie.

        # However, zodbupdate assumes that the initial value for

        # `next` is interpreted like FileStorage does: as an OID (8

        # bytes) and uses that to specify an object id to start

        # with. (inclusive)

        #

        # We need to detect the case that we're given an OID; that should only happen

        # in the first call.

                "There is no defined value for the *next* parameter. "

                "RelStorage will rely on implementation-defined behaviour and treat it like "

                "FileStorage does, while assuming that iteration is beginning from the start. "

                "This may change in the future."

            )

            # Beginning.

            # There *should* always be at least one object: the root.

            # But if they passed a non-0 ``next`` value, that's not guaranteed.

            # However, the protocol is not very good at expressing that possibility:

            # The typical call looks like:

            #    oid, tid, data, next = storage.record_iternext(next)

            # Unpacking will raise if we return None; most processing of *data* is not expecting

            # None (it's common to wrap it in ``io.BytesIO()`` immediately; that's what zodbupdate

            # does). Thus, it's probably probably best to let getting the first value simply raise.

            # After that, we can treat it like it was given to us in *next*

        # Somewhere in the middle, possibly the end.

        #

        # We have to operate one ahead (keep one buffered) because

        # we need to be able to tell the caller not to call back.

            # Signal not to call back.

        else:

        # Note that this method exists mainly to deal with read-only

        # transactions that don't go through 2-phase commit (although

        # it's called for all transactions). For this reason, we only

        # have to roll back the load connection. (The store connection

        # is completed during normal write-transaction commit or

        # abort.)

        # The next time we use the load connection, it will need to poll

        # and will call our __on_first_use.

        # Typically our next call from the ZODB Connection will be from its

        # `newTransaction` method, a forced `sync` followed by `poll_invalidations`.

        # As with `poll_invalidations`, the cache needs to know about transaction

        # boundaries like this to optimize polling for changes and invalidations.

        # If we were a write transaction, that's already done by the call to ``after_tpc_finish``,

        # but for a read-only transaction, it wouldn't happen. So for this reason,

        # as with `poll_invalidations`, the cache is in charge.

        # This doesn't use restart_and_call() or even just restart() because we don't

        # need to do those checks yet, we just want to quietly rollback.

        # They both rollback; the difference is that restart_load checks for replicas,

        # and calls any hooks needed.

        """

        Updates to a current view of the database.

        This is implemented by rolling back the relational database

        transaction.

        .. versionchanged:: 3.0a6

           This method now pays attention to the *force* argument. If it is

           ``False``, this method does nothing. This prevents extra rollbacks

           and network traffic in the common case of using the default implicit

           transactions when this object is used by a ZODB ``Connection``.

        """

            # When we get a force=False, it's from Connection.afterCompletion()

            # calling Connection.newTransaction() because the transaction manager is

            # in implicit mode. Immediately after it does that, it will call

            # poll_invalidations()...and in our implementation, that implies a sync.

            # So we can avoid the overhead of the extra rollback.

            # Disconnected. Well, the rollback happens automatically

            # in that case. No big deal, ignore it.

            #

            # However, if we happened to be in the middle of committing and were

            # asked to sync (XXX: why would we?) then that's probably a problem that

            # needs to be handled and the commit rolled back too.

        """

        Look for OIDs of objects that changed since _prev_polled_tid.

        Returns {oid: 1}, or None if all objects need to be invalidated

        because prev_polled_tid is not in the database (presumably it

        has been packed).

        """

        # poll_invalidations is called by Connection.newTransaction(), which is a

        # transaction synchronizer method.

        # Connection.afterCompletion() also calls Connection.newTransaction() if we're not

        # using explicit transactions.

        # Our sync() will have been called just before this, so we should not need to

        # restart the connection, just make a call. The load_connection logic handles that

        # for us.

            # If we've never loaded (self._load_connection is false),

            # it might seem that there's no point in doing a poll, but

            # that's not true. testMVCC breaks if we skip this. It's not exactly clear

            # why.

        # __on_load_first_use is also the function we gave the LoadConnection to automatically

        # call when we access the cursor for the first time. In this situation, we optimize

        # for when the function is the same and only call it once.

            self.__on_load_first_use

        )

        # Now we're in a fully synced state, meaning

        # we don't need to do anything fancy with the load connection

        # anymore. Let it know that.

        else:

            # The Connection doesn't care about this, it just passes it

            # to the PickleCache. The Python implementation of the PickleCache

            # takes any iterable of oid bytes, but the C implementation deals only

            # with an actual dict(), or something that is a sequence and can be

            # iterated using integer indices. If you give it a dict, all it cares

            # about are the keys.

        # Reset the Connection cache. This makes it more likely we'll be called to

        # load an object and the application can notice the issue.

        # Allow GC to do its thing with the locals, don't hold them indefinitely

        # The next time we rollback or reopen, we're no longer stale

        # Note that this creates a reference cycle, but it should only be

        # temporary.

        # This is called at the end of every transaction in

        # afterCompletion(), and at the beginning of every transaction

        # in sync() (called from Connection.newTransaction(), which is

        # called automatically from Connection.afterCompletion when

        # transactions are not explicit) Most of the time we won't be

        # stale, which is why we only install these hooks when we are,

        # and are careful to clean them up when we're not.

        #

        # Both _conn and _cursor will be None if the rollback detected

        # a disconnect from the database.

            k: v.no_longer_stale()

            for k, v in my_ns.items()

            if callable(getattr(v, 'no_longer_stale', None))

        }

        """

        Poll for invalidations, update our cache.

        Move the component objects of this object into or out of the

        stale state as appropriate.

        Primarily in tests which directly use storage APIs, this may

        be called automatically, between transactions, when a

        Connection is not in use to receive invalidations. For that

        reason, we queue changes seen here as we keep moving forward

        and can apply them all when a connection asks us.

        """

        # Ignore changes made by the last transaction committed

        # by this connection, we don't want to ghost objects that we're sure

        # are up-to-date unless someone else has changed them.

        # Note that transactions can happen between us committing and polling.

        else:

        # get a list of changed OIDs and the most recent tid

                conn, cursor,

                ignore_tid

            )

            # The database connection or cache is stale, but postpone

            # this error until the application tries to read or write

            # something and is better equipped to deal with it. (This

            # is called from transaction.begin() ->

            # Connection.newTransaction ->

            # RelStorage.poll_invalidations, and no one is prepared

            # for transaction.begin() to fail, or to handle it

            # gracefully if it does.)

            # If we were configured to revert-when-stale, we wouldn't

            # get a ReadConflictError, we'd have a return of None and

            # our cache would restart polling. Eventually the RCE

            # should stop as the replica catches up. But if we clear

            # the caches on the RCE, they'll lose their polling state

            # and essentially revert-when-stale even though we weren't

            # configured to do so.

            # At a minimum, though, we must reset our Connection's

            # object cache. If we fail do do this, then we see issue

            # in the zodbconvert tests, when the destination storage

            # configuration loaded to do the conversion and zapping on

            # didn't have matching persistent cache information as the

            # storage configuration that had initially populated the

            # destination (in order to verify zapping). As our cache

            # implementation got better and better, we suddenly found

            # that without zapping the cache files, we would report

            # this RCE, based on info from the persistent cache from

            # before the conversion, with no way to fix it (short of

            # zapping those cache files). This can also come up with

            # packing in history-free databases

            # (checkPackAllRevisions). We have a temporary fix for

            # that in self.pack() until we write more specific tests.

            #

            # TODO: We need a way to distinguish RCE from packing

            # vs RCE from a replica switch.

            # A CCE, on the other hand, means the cache has already

            # reset its own internal state, we outght to do the same for

            # the Connection. See above for why this shouldn't

            # propagate.

            # Stop tracking invalidations and reset the ZODB object cache.

            # This is reset by poll_invalidations.

                # Hmm, ok, the Connection isn't polling us in a timely fashion.

                # Maybe we're the root storage? Maybe our APIs are being used

                # independently? At any rate, we're going to stop tracking now;

                # if a Connection eventually gets around to polling us, they'll

                # need to clear their whole cache

        """Return a directory that should be used for uncommitted blob data.

        If Blobs use this, then commits can be performed with a simple rename.

        """

            # In a history free database, it's *possible*

            # that the database's last transaction ID could now have actually

            # gone backwards! I strongly suspect this only happens in fabricated

            # test scenarios (e.g., the last transaction in the database is an object that

            # was directly stored without any connection to the object graph starting from the