cloudmarker / cloudmarker / 322

84%

push

travis-ci-com

Create raw, ext, and com top-level buckets

In order to avoid polluting the raw data obtained from clouds with
additional metadata, this change separates the data in records into
three buckets:

  - raw: to store the raw data obtained from the clouds
  - ext: to store extended metadata specific to each cloud
  - com: to store common metadata applicable to all clouds

This change also increases the degree of decoupling between various
plugins. The store plugins no longer depend on the `record_type`
attribute populated by the cloud plugins. The store plugins take the
entire data, treat the data as unstructured (schemaless), and write the
entire data as is to the destination stores.

For example, the MongoDBStore plugin no longer stores the data in
separate MongoDB collections named by record types. All records go into
a single collection. As a result, schema validation in the MongoDBStore
plugin has also been removed.

The FileStore plugin though still maintains separate files named by
origin worker names, i.e., the name of the worker that created the
record. We decided not to dump all the records into a single file
because then we would have to implement file locking and carefully
maintaining correct JSON syntax (opening and closing square brackets)
while multiple instances of FileStore in multiple workers try to write
to the same file.

Since the FileStore plugin does not depend on record type anymore,
FileStore too is decoupled from other plugins. The origin worker name is
populated in `record['com']['origin_worker']` by the workers module,
i.e., the framework part of the code, so FileStore is slightly coupled
with the framework but completely decoupled from other plugins. This
slight coupling with the framework is not a major concern because
FileStore is an experimental plugin for testing purpose only.

70 of 103 branches covered (67.96%)

Branch coverage included in aggregate %.

448 of 570 relevant lines covered (78.6%)

2.36 hits per line