Kapiche / caterpillar / 177
100%
master: 100%

LAST BUILD BRANCH: 0.15.10
DEFAULT BRANCH: master
Ran
Jobs 1
Files 0
Run time –
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
177

push

travis-ci

SamHames 
Change serialisation format

Updates the data model for storing the index to better take advantage of SQLite
by providing a richer relational data model. Improves performance by avoiding
the need to deserialise, update and reserialise large components of the index
in order to update the terms.

API Changes

- ID's (plugin/frame/documents) are monotonically increasing integer ID's
assigned during commit: assigned identifiers are available as attributes of
the IndexWriter object after commit() has been called. This is both simpler
and faster, because the writer does not need to coordinate with the index
until commit time and safer, because ID's are not available until the data is
sucessfully committed.

- The character level word boundaries (positions) is stored at the term level
rather than on the frame. The frame level '_positions' field is still
available, but is not updated with merge_terms or bigrams.

- QueryResults now collect term-frequency information as they collect matching
frames.

- Fold case and merge terms operations need to be conducted in separate
transactions from other write operations.

- Index revisions can be identified by a monotonically increasing counter with
details of the number of changing documents. If document B is added after
document A it will always have a higher document id. Document ID's are not
reused (except for a potential update_document operation to be added later).

Notes

Much of the heavy lifting is now deferred to the storage objects - these act as
file format specific layers that are aware of the data layout and performance
implications. The Index* objects focus on providing API rather than storage
logic.

The low level SqliteStorage object is now a thin wrapper over two distinct
reader/writer classes. The writer object now maintains an in memory SQLite
database to cache updates until .commit() is called. When commit is called  an
SQL script is run, allowing for faster ingestion of data. The write lock is not
strictly needed until commit is called.

The SQLiteReader class exposes a more sophisticated field selection interface,
but this is not yet available at the Index level for backward compatibility.

The associations matrix is no longer generated during indexing but created on
demand from the posting tables. This makes indexing faster at the cost  of
slower calculation of the associations matrix. This kind of calculation  would
be necessary anyway for query specific association matrix calculation anyway.

The core posting tables are available in two sort orders: term-frame for
searching by term and frame-term to allow frame level term-frequency vectors to
be provided. Character positions are available at the individual term level.
Jobs
ID Job ID Ran Files Coverage
1 177.1 0
Travis Job 177.1
Source Files on build 177
Detailed source file information is not available for this build.