MITLibraries / oastats-backend / 159

Builds Branch Commit Type Ran Committer Via Coverage
159 roast Add load command to dump mongo into postgres This adds a load command that will take the requests data currently in mongo and dump it into the new postgres data model. This is one-use functionality that can be removed once the migration is complete. push gravesm travis-ci pending completion   set done
158 roast Fix handle summary query push gravesm travis-ci pending completion   set done
157 roast Fix mongo collection access push gravesm travis-ci pending completion   set done
156 roast Use stricter date validation Some of the dates are malformed in the logs and getting through the lazy match. arrow doesn't seem to care, so this change should filter out the messed up dates. push gravesm travis-ci pending completion   set done
155 roast Add summary command This adds full support for generating the summary collection from postgres. push gravesm travis-ci pending completion   set done
154 roast Add support for database config from env var The command line will now pull the database URI from an environment variable called `OASTATS_DATABASE` if available. push gravesm travis-ci pending completion   set done
153 roast Add logging config push gravesm travis-ci pending completion   set done
151 roast Add error handling in pipeline This commit addresses the various errors that can occur due to bad data entering the pipeline. Errors are logged as warnings, the offending request is dropped and processing continues. push gravesm travis-ci pending completion   set done
150 roast Add output to CSV for PostGres COPY This adds a CSV formatted output suitable for passing to the PostGres COPY command. Only the requests table is populated in this way; the underlying documents and identity tables get updated during the run. Thi... push gravesm travis-ci pending completion   set done
149 roast WIP push gravesm travis-ci pending completion   set done
148 roast Add document insertion into postgres This commit provides the framework for inserting documents into postgres. A memory cache is provided through dogpile to improve performance while running over lots of requests. After some time most documents, ... push gravesm travis-ci pending completion   set done
147 roast Refactor OAStats pipeline This is a significant refactoring of the pipeline in preparation for moving to Postgres. Much of the code has been simplified and removed since we no longer need to deal with adding a summary collection. Some further opt... push gravesm travis-ci pending completion   set done
145 roast WIP push gravesm travis-ci pending completion   set done
144 master Merge pull request #56 from MITLibraries/config Remove pipeline configuration from file push gravesm travis-ci pending completion   set done
143 master Remove pipeline configuration from file This removes the pipeline config file in favor of configuring from the command line. Command line config will be easier to handle at runtime. Pull #56 gravesm travis-ci pending completion   set done