alphagov / backdrop

91%

master: 89%

LAST BUILD ON BRANCH fix-missing-day-dvsa-takeup

branch: fix-missing-day-dvsa-takeup

CHANGE BRANCH

x

Reset

• fix-missing-day-dvsa-takeup
• 159387416-redis
• 404-for-bucket-not-found
• 80723696-iterate-the-flattening
• abort-on-error
• accept-excel-files
• add-ack-exclude
• add-build-to-ackrc
• add-deprecation-warning
• add-docs-on-emptying-dataset
• add-flatten-param
• add-not-authorized
• add-ops-manual-documentation
• add-patch-to-write-api
• add-procfile-worker
• add-production-config
• add-quality
• add-quarter
• add-redirect-test
• add-request-details-to-response-log
• add-service-id
• add-service-id-to-dataset
• add-statsd-prefix
• add-test-perf-dir-to-ackrc
• add-warning-to-upload
• add_config_for_electronic_vehicle_licensing_monitoring
• add_dataset_name_to_error_messages
• add_flower
• add_rediss_support
• add_tags_to_json_logger
• add_timers
• admin-caching
• admin-improve-ux
• allow-collecting-with-period-queries
• allow-expected-time-to-be-updated
• allow-saving-none
• allow-special-field-names
• allow-travis-fail
• allow-unknown-data-set-fields
• always-utc-out
• always_add_a_default_upload_filter
• audit-logging
• auth_headers_the_second
• authentication-responses
• automate-creation-of-buckets-55989156
• automate-user-creation
• back-to-bucket-list
• backdrop-errors
• backdrop-stagecraft-integration
• better-error-handling
• better-upload-error-messages
• browser-screenshot-on-failure
• bucket_recency
• bugfix-evl-dodgy-spreadsheet
• bugfix-flat-data-shifting
• bugfix-redirect-root-to-admin
• bugfix-transformer-rate
• bugfix/add-test-for-empty-data-with-duration-parameter
• bugfix/no-data-transforms
• bugfix/spotlight-cors-request-headers
• bugfix/user-sat-transform
• bump-client-version
• bumping-selenium
• cache_aff
• change-dev-to-match-other-envs
• change-flattened-response-format
• change-logstash-library
• change-production-log-location
• changes-for-govuk-devvm
• check_upload_mimetypes
• clamav-errors
• clamdscan
• clearer-content-type-error
• click-jacking
• consistent-errors-single-commit
• consistent-write-errors
• consistent-write-errors-refactor
• convert-excel-float-to-int
• convert-timestamp-to-utc-in-id
• cors-on-error
• create-ackrc
• create-collection-endpoint
• create-transform-data-set
• crossdomain
• data-migration
• day-and-hour-period
• decouple-collection-create
• decrease-graphite-disk-usage
• delete-data-set-record
• delete-returns-200
• delete_datasets
• dependency-updates
• deployed-to-dev
• deployed-to-integration
• deployed-to-preview
• deployed-to-production
• deployed-to-staging
• disable-file-upload-scenario
• disable-travis-emails
• do-not-verify-stagcraft-host
• do_not_go_gentle_into_that_rewrite
• docker-compose
• document-write-api
• dont-overwrite-virtualenv
• drop-collections-on-replicate
• dump-buckets-tool
• dump_help
• emit-write-messages
• empty-data-set-endpoint
• encode-unicode-strings-in-record
• ensure-only-full-satisfaction-weeks-transformed
• evl-channel-volumetrics-totals
• evl-customer-satisfaction-upload
• evl-data-upload
• evl-satisfaction-raw-queries
• evl-volumetrics
• evl-volumetrics-upload
• explain_how_to_run_a_manual_transform
• feature-flags
• feature/add-prefix-filter
• feature/add-regex-filter
• feature/aggregate-data-parity
• feature/architecture-notes
• feature/autopep
• feature/better-ranges
• feature/cache-control
• feature/data-for-transforms
• feature/delete-unused-code
• feature/dependency-upgrades
• feature/dry-auth-checks
• feature/dry-request-id
• feature/dry-staleness-check
• feature/exclude-tagged-branches-from-travis
• feature/gzip-bomb
• feature/informative-adr-names
• feature/log-request-id
• feature/mapping-transform
• feature/more-detailed-error-message
• feature/perf-decorators
• feature/rate-limiting
• feature/rate-transform
• feature/required-fields-for-transforms
• feature/signon-client-id
• feature/time-all-the-things
• feature/transform-improvements
• feature/update-splinter
• feature/upgrade-dependencies
• feature/upgrade-requests
• feature/write-start-end-at
• fill-data-perf-improvments
• filter-latest-by-record-id
• filth
• fix-bearer-token-handling
• fix-behave-new-phantom
• fix-curl-in-readme
• fix-decompressing-request
• fix-dev-stagecraft-token
• fix-evl-upload-failure
• fix-field-names
• fix-flattened-shifting
• fix-ga-bucket-timezones
• fix-ids
• fix-integer-auto-id-error
• fix-migrations
• fix-race-condition
• fix-replicate-db
• fix-replicate-script
• fix-route-name
• fix-stagecraft-port
• fix-status-error
• fix-tests-for-emptying
• fix-the-build
• fix-urls
• fix-valid_start_at
• fix/audit-log-token
• fix/audit-nonempty-appends
• fix/latest-data-query-params
• fix_auto_ids_on_timestamps
• fix_bucket_access
• fix_celery_config_write_api
• fix_replicate_script
• fixing-file-upload
• flaskutils-pep8
• flatten-response
• force-settings-update
• force-utc-timestamp-in-ids
• format-push-api-date-timestamps-utc
• get-bucket-config-from-stagecraft
• get-clamdscans-stderr
• get-last-data-even-if-none
• get-latest-tx-data
• get_latest_but_go_back_in_time_on_nulls
• gitignore-swap-files
• go-back-in-time-for-data-again
• hack-timestamps
• handle-multiple-dashboards-for-service-id
• http-put-to-empty-data-set
• idempotent-migration
• ignore-collections-in-replicate
• ignore-specified-tx-data-types
• improve-db-migrations-instructions
• improve-delete-test
• improve-error-messages-1
• improve-error-messages-4
• improve-error-messages-5
• improve-naming-convention
• instrument-transforms
• integrate-with-coveralls
• integrate-with-coveralls-clean
• invalidTokenMessage
• is_travis_selenium_slow
• jenkinsfile
• json-to-csv-convert-script
• link-to-manual-entry
• listen-to-signonotron
• load-seeds-from-app-directory
• log-to-stdout-in-development-mode
• logstash_json
• master
• master-stagecraft
• mattbostock-patch-1
• max_age_expected
• migrate-buckets-to-data-sets
• migrations-by-pattern
• migrations-use-repository
• monitor-gcloud-sales
• more-helpful-error-message
• more-robust-start-app
• more-robust-subclassing
• more_log_response_messages
• move-past-incomplete-periods
• move-query-response-to-core
• move-tools-into-tools
• multiple-group_by-params
• new-data-set
• no-pep-on-build
• no-users-on-write
• nocd-on-replicate-script
• nosetests-xml-ackrc-ignore
• only-allow-utc-timestamp-in-ids
• only-run-migration-if-not-capped
• paas-migration
• parse-csv-numbers
• pep8
• perf/data-set-status
• pipper
• point_dev_uploader_to_new_url
• postgres-storage
• production
• production-log
• profile-writes
• protect-against-empty-dashboard-list
• push-data-script
• py26-compliance
• quickstart
• quieten-pip
• ratty-http
• read-api-collect-maths
• read-data-set-format-csv
• read2core
• really_reliable_replicated_replicas
• recent-last-api
• refactor-behave-tests
• refactor-data-layer-step-1
• refactor-data-layer-step-2
• refactor-data-layer-step-3
• refactor-data-layer-step-4
• refactor-data-layer-step-5
• refactor-run-tests
• refactor-use-oauth-tokens
• refactor-write-api-functions
• relative-time-api
• release-backup
• release_1012
• release_1013
• release_1014
• release_1015
• release_1016
• release_1017
• release_1019
• release_1021
• release_1022
• release_1023
• release_1026
• release_1027
• release_1028
• release_1029
• release_1030
• release_1031
• release_1032
• release_1033
• release_1034
• release_1035
• release_1036
• release_1037
• release_1038
• release_1039
• release_1040
• release_1041
• release_1042
• release_1043
• release_1046
• release_1047
• release_1048
• release_1050
• release_1051
• release_1052
• release_1053
• release_1054
• release_1055
• release_1056
• release_1057
• release_1058
• release_1059
• release_1060
• release_1061
• release_1062
• release_1064
• release_1065
• release_1066
• release_1067
• release_1068
• release_1069
• release_1070
• release_1072
• release_1075
• release_1077
• release_1078
• release_1089
• release_1094
• release_1095
• release_1102
• release_1103
• release_1106
• release_1108
• release_1109
• release_1110
• release_1111
• release_1112
• release_1116
• release_1117
• release_1118
• release_1119
• release_1120
• release_1122
• release_1123
• release_1124
• release_1125
• release_1126
• release_1127
• release_1128
• release_1129
• release_1130
• release_1131
• release_1132
• release_1133
• release_1134
• release_1137
• release_1138
• release_1139
• release_1140
• release_1141
• release_1142
• release_1143
• release_1144
• release_1145
• release_1146
• release_1147
• release_1148
• release_1149
• release_1150
• release_1151
• release_1152
• release_1153
• release_1154
• release_1155
• release_1156
• release_1157
• release_1158
• release_1159
• release_1160
• release_1161
• release_1162
• release_1163
• release_1164
• release_1165
• release_1166
• release_1167
• release_1168
• release_1169
• release_1170
• release_1171
• release_1172
• release_1173
• release_1174
• release_1175
• release_1176
• release_1177
• release_1178
• remove-STAGECRAFT_DATA_SET_QUERY_TOKEN
• remove-admin-app
• remove-ambiguous-letters-from-tokens
• remove-create-route
• remove-example-csv
• remove-latest-data-restriction
• remove-mongo-user-repository
• remove-nested-grouped
• remove-old-buckets-legacy
• remove-old-procfiles
• remove-python-26
• remove-redundant-code
• remove-signon-flask-dependency
• remove-unused-bits
• remove-updated-at-buckets
• remove-validation-collect-internal-fields
• rename-buckets-to-data-sets
• rename-hosts-var
• rename_bucket_to_data_set
• reordering-setup-in-docs
• replica-set-config-for-dev-vm
• replica_set_as_variable
• replicate-db-fail-earlier
• replicate-db-script-vm-option
• request-decompression
• request-id-govuk
• return-user-friendly-date-error-messages
• revert-333-status_json
• revert-450-only-allow-utc-timestamp-in-ids
• rotating-logs
• run-tests-blow-up-sooner
• run-travis-on-master
• sanity-check-1
• scan-files-on-upload
• schema_validation_101
• schema_validation_102
• send-auth-token-to-stagecraft
• send-auth-token-to-stagecraft2
• separate-admin-app
• serialize-dev-startup
• service-page-latest-data-transform
• set-failover
• set-travis-mongo-rs
• set_2weeks_capped_size
• set_2weeks_capped_size_2
• shorter-tokens
• show_failing_bucket_times
• signonotron2-integration
• simplify-conditional
• skip-blank-values
• skip-splinter-tests
• split-up-write-api-feature-tests
• stagcraft-integration-single-test-diff
• stagecraft-integration
• staging
• staging-2018-08-29
• start-up-documentation
• status_json
• step_5_test
• subgroup_keys
• sync-to-govuk-dev-vm
• temp-new-query-repo-automate-creation-of-buckets
• test-behave-travis
• test-relative-date-params
• tidy-nested-merge
• tidy_manifest
• transaction-explorer-transform
• transactions-explorer-transform-clean-hist
• transform-endpoint-tests
• transformations
• transforms-endpoint
• travis-containerise
• travis-experiments
• treat-blank-cell-as-null
• try-new-splinter
• unicode-characters-transactions-explorer
• unicode-is-hard
• unpublisheddisclaimer
• update-amqp-url-for-govuk-dev-vm
• update-db-location
• update-dev-stagecraft-urls
• update-dev-transform-config
• update-documentation
• update-last-time-expected-migration
• update-perfplat-client
• update-readme-rabbitmq-dev-set-up
• update-readme-replicatedb
• update-readme-with-worker-info
• update-replicate-script
• update-testrunner
• update-upload-page-copy
• update-url-for-govuk-dev-vm
• update-write-validations
• update_last_time_expected_migration
• upgrade_paas_stack
• upgrade_splinter
• upload-validation
• use-postgres-159387515
• use-urandom-not-random
• use-write-api
• useful-scripts
• user-satisfaction-task
• users-from-stagecraft
• volumesInSubGroups
• wat
• well_formed_auth_headers_bruv
• wercker
• write-deployed-to-dev
• write_api_accept_type_and_group

Query for latest data inclusively.

Currently when a spreadsheet of data with timestamps between, say, the 26/02/2015 and
01/02/2015 this represents 7 days. However, the inferred start and end
ats inferred from this for the subsequent transform are the 26th to the
1st. As these are interpreted exclusively by backdrop it will only get 6
days of data back until the next upload. If the next upload contains the
data for this first week again and the next week - 02/02/2015 to
07/02/2015 it will then perform another exclusive query for the 26th to
the 7th. This time though it will include the FULL first week and update the
existing data accordingly, however the second week will again be one day
short. This ensures that dates passed through from
the write api are interpreted inclusively which seems like the expected
behaviour.