cloudmarker / cloudmarker

77%

master: 84%

LAST BUILD ON BRANCH buckets

branch: buckets

CHANGE BRANCH

x

Reset

• buckets
• 0.1.0
• alertdocs
• apidocindent
• audit-version
• auditemails
• azcloud
• azstorage
• azurecloud
• azurecloud-exception
• azvm_record_type
• azwebapp
• badges
• baseconfig
• bracket
• build-matrix
• changelog
• clioptions
• clouderror
• common-plugins-config
• consistency
• copy-ext
• copyext
• cov_clean
• dependabot/pip/cryptography-3.3.2
• dependabot/pip/httplib2-0.18.0
• dependabot/pip/httplib2-0.19.0
• dependabot/pip/pyyaml-5.4
• dependabot/pip/rsa-4.7
• dependabot/pip/urllib3-1.26.5
• dist
• docfixes
• doctest-fix
• email_debug_param_from_config
• emailbody
• emaildocs
• empty-port
• esstore
• firewalleventtest
• firewallnorm
• firewallruleevent
• fix-esstore-done
• fix-unused-input-queue
• fix_cloud_worker
• fixbuild
• fixcoverage
• fixindent
• freeze
• gcpcloudlogs
• gcpcloudtest
• iam_policy
• ignore-virtual-machines
• ioworkers
• key_file_name_fix
• lint
• load_yaml_safely
• log
• logfixes
• logger-defaults
• logging-levels
• master
• maxrecs
• mock-records
• mockplugins
• mongodbdefaults
• mongodbfix
• mongodbtests
• multiprocessing
• pluginframework
• pluginloader
• readable-version
• readme
• remove-email-config
• removefirewallevent
• rename_logger
• renameparams
• requirements
• rest
• reuse-mock
• rm-venv
• slackalert
• smtpssl
• sphinx
• sphinx-rst-update
• splunkfixes
• splunkhecstore_docs
• splunkhectokentest
• splunkstore
• sub_id_rename
• tests
• travis
• userconfig
• userdoc
• userecordterm
• verdev
• webapp
• worker-com-fields
• worker-error-handling
• workerlogs

Build # 321

Build Type

Pull #46

travis-ci-com

Committed by web-flow

Commit Message

Create raw, ext, and com top-level buckets

In order to avoid polluting the raw data obtained from clouds with
additional metadata, this change separates the data in records into
three buckets:

  - raw: to store the raw data obtained from the clouds
  - ext: to store extended metadata specific to each cloud
  - com: to store common metadata applicable to all clouds

This change also increases the degree of decoupling between various
plugins. The store plugins no longer depend on the `record_type`
attribute populated by the cloud plugins. The store plugins take the
entire data, treat the data as unstructured (schemaless), and write the
entire data as is to the destination stores.

For example, the MongoDBStore plugin no longer stores the data in
separate MongoDB collections named by record types. All records go into
a single collection. As a result, schema validation in the MongoDBStore
plugin has also been removed.

The FileStore plugin though still maintains separate files named by
origin worker names, i.e., the name of the worker that created the
record. We decided not to dump all the records into a single file
because then we would have to implement file locking and carefully
maintaining correct JSON syntax (opening and closing square brackets)
while multiple instances of FileStore in multiple workers try to write
to the same file.

Since the FileStore plugin does not depend on record type anymore,
FileStore too is decoupled from other plugins. The origin worker name is
populated in `record['com']['origin_worker']` by the workers module,
i.e., the framework part of the code, so FileStore is slightly coupled
with the framework but completely decoupled from other plugins. This
slight coupling with the framework is not a major concern because
FileStore is an experimental plugin for testing purpose only.

Pull Request Pull Request #46: Create raw, ext, and com top-level buckets

Run Details

70 of 103 branches covered (67.96%)

Branch coverage included in aggregate %.

58 of 58 new or added lines in 8 files covered. (100.0%)

448 of 570 relevant lines covered (78.6%)

2.36 hits per line