tarantool / tarantool / 6065

45%

master: 87%

push

travis-ci

vinyl: store metadata in memtx

Instead trying to encode information about vinyl runs in the file
layout, let's store it in a memtx table with the following schema:

  [ run_id, space_id, index_id, index_lsn, state, begin, end ]

where

  server_id - ID of the server that created this run
  run_id - run ID, unique system-wide
  space_id - ID of the space this run is for
  index_id - ID of the index this run is for
  index_lsn - LSN at the time of index creation
  state - run state (see below)
  begin, and - range boundaries

server_id and run_id together constitute the primary key.

Run state is one of:
 - Committed. Run was created and must be recovered. Run file may be
   absent though, in which case we create an in-memory index on recovery
   instead of the run.
 - Deleted. Run was deleted by compaction. Needed for garbage
   collection.
 - Reserved. ID is reserved for a run which is going to be produced by
   compaction. If compaction succeeds, transformed to 'committed',
   otherwise to 'failed'.
 - Failed. Run left from failed compaction.

Old run files are deleted and the table is cleaned on successful
checkpoint.

Note, a record is created for a run before it makes its way to disk: on
dump we create a record for the *new* in-memory index, while the record
corresponding to the in-memory index scheduled for dump must already be
present (created by previous dump of the same range). On recovery we
detect whether a run was dumped or not by checking presence of the
corresponding file on disk. If the file is there, we simply skip xlog
records going to the range.

This approach has an inherent problem when it comes to range splitting
or coalescing. The thing is we don't dump in-memory indexes separately
when splitting or coalescing a range - instead we compact them all in a
new run. As a result, on recovery we won't find the file corresponding
to an in-memory index of the range that was split and hence will have to
replay xlog records going to the ran... (continued)

558 of 558 new or added lines in 5 files covered. (100.0%)

25354 of 33413 relevant lines covered (75.88%)

866124.16 hits per line