airbnb / streamalert / 4793

95%

push

travis-ci

rule processor function split to classifier and rules engine (#843)

* function split part 1, migration and refactor of payload and parsers (#821)

* start of new classifier function with payload breakout

* big revamp of parser classes

* some additional parser updates and simplification

* completed refactor of parsers, including more thorough and accurate/valid testing

* adding updated payload tests and fixing bugs

* major s3 payload updates to support all types of data such as:

- new line delimited data
- raw json objects (non-newline)
- gzipped files (both new line and non)
- regular files (both new line and non)

* adding tests for all payload types

* updates to requirements to remove jsonpath_rw and add jsonlines/jmespath

* misc other updates

* updates to docstrings

* failing hard on log_pattern bad key name

* addressing PR feedback

* addressing PR feedback

* function split part 2, adding classifier and normalizer  (#822)

* update parser property name to be more clear

* adding `Normalizer` class to handle log normalization

* adding tests for the Normalizer class

* make some properties on the parser class protected so usage is clearer

* in order to reduce confusion about what properties should be publicly 
accessed from this class, I've made all properties that are only used 
internally protected and not public

* adding `Classifier` class and making some changes to payload

* adding new conf/types_new.json file to use a more expressive format

* updating StreamPayload base class tests

* adding tests for PayloadRecord

* some updates to classifier and payload base class

* adding `Classifier` tests and updating payload tests

* removing trailing spaces in types_new.json

* addressing PR feedback

* function split part 3, adding firehose client for data retention (#823)

* adding shared helper to return default boto config with timeouts, etc

* adding historical da... (continued)

1242 of 1242 new or added lines in 32 files covered. (100.0%)

4358 of 4483 relevant lines covered (97.21%)

0.97 hits per line