UrbanCCD-UChicago / plenario2

76%

master: 76%

LAST BUILD ON BRANCH 344-ingest-errors

branch: 344-ingest-errors

CHANGE BRANCH

x

Reset

• 344-ingest-errors
• 0.12.4-debug
• 0.2.13
• 0.4
• 0.5
• 216-accept-204
• 221-add-back-shapefile
• 222-field-guesser
• 264-chart-metas
• 264-graph-metas
• 330-display-graphs
• 331-maps
• 338-fix-sentry
• 348-adios-explode
• 350-clean-up-etl
• 362-index-names
• 374-chart-rendering
• 378-user-request-changes
• 383-chart-actions
• 390-fix-seeds-and-index-names
• 395-fix-erl-to-ndt-cast
• 398-api-pagination
• 421-document-api
• 4xx-handling
• add-description-to-data-set-field
• add-dsfields-to-meta-detail
• add-sentry
• add-work-to-scheduler
• admin-start
• admin-user-workflow
• admin-workflow-meta-approval
• aot-api
• aot-explorer
• aot-redo
• api-error-handling
• api-purge
• assert-emails-are-sent
• bbox
• better-401-403
• better-404
• better-500
• better-admin
• better-maps
• box-filter
• build-deploy-updates
• build-tag
• ci-cd
• clean-up-55
• clean-up-user-actions
• contributor-docs
• controller-tests
• coverage
• data-set-actions
• data-set-actions-redo
• data-set-redo
• dev-seeds
• disable-meta-edit
• dockerignore
• documentation
• drop-unique-constraints
• dropping-the-deuce
• etl
• etl-datetime-parsing
• etl-fsm
• etl-redo
• etl-redo-2
• etl-schedule-fixer-upper
• etl-scheduler
• etl-set-meta-props
• expand-ingest-timeout
• export-emails
• export-encodings
• export-logs
• export-search-results
• exports
• field-inspection-limits
• fix-build-script
• fix-deployment
• fix-explorer-strftime-error
• fix-field-guessing
• fix-mixed-srids-179
• fix-parse-geoms
• fix-range-query
• fix-static-assets
• fix-to-erl
• fix-typo-seed-script
• frontend
• frontend-refactor
• frontend-restructuring
• hoek-vulnerability-fix
• https
• info-blocks
• issue-203
• issue-204
• issue-223
• issue-224
• issue-225
• issue-226
• issue-231
• issue-233
• issue-236
• issue-244
• issue-245
• issue-246
• issue-257
• issue-310
• issue-353
• jenkins
• jenkins-start
• json-error-responses
• leaflet-marker-hotfix
• list-endpoint
• logo
• master
• meta-add-data-set-fields
• meta-bbox
• meta-fsm
• meta-mutation-requests
• meta-ready-check
• model-reg-types
• momemnt-cve
• move-sql-temps
• nix-foo-bar-baz
• o-canada
• one-point-x-frontend
• one-six-formatter
• options-response
• page-param-errors
• pagination
• pagination-limit
• peer-key-error
• plenarex-merge
• port-comparisons
• port-search-component
• pretty-search-results
• query-utils
• readme-hotfix
• readme-test-info
• redo-admin-user-notes
• refactor
• refactor-auth
• refactor-detail-template
• register-dataset
• register-user
• registry
• reinstate-travis
• release-migrate-cmd
• reorg
• rm-hexfaktor
• s3-bucket-configuration
• scheduler-fix
• seeds-hotfix
• self-management
• send-email
• shim
• shim-endpoint-response-formatting
• shim-keywords
• shim-limit-offset-keywords
• slug-endpoint
• spruce-up-explorer
• stable-pagination
• stream-etl
• swap-n-drop
• tempfiles
• test-constaints
• test-etl-jobs
• test-export-job
• test-fields
• test-init
• test-init-formatted
• test-meta
• tool-versioning
• tooltip-icons-not-rendering
• ui-tweaks
• umbrella
• update-queries
• user-auth
• v.0.13.1
• v0.10.1
• v0.11.0
• v0.11.2
• v0.11.5
• v0.12.0
• v0.12.1
• v0.12.2
• v0.12.3
• v0.12.4
• v0.12.4-debug
• v0.12.5
• v0.13.0
• v0.13.1
• v0.13.2
• v0.13.3
• v0.13.4
• v0.13.5
• v0.13.6
• v0.13.8
• v0.14.0
• v0.2.10
• v0.2.11
• v0.2.12
• v0.2.15
• v0.2.9
• v0.5.0
• v0.6.0
• v0.6.1
• v0.6.2
• v0.6.3
• v0.6.4
• v0.7.0
• v0.7.1
• v0.7.2
• v0.7.3
• v0.7.5
• v0.8.1
• v0.9.0
• v0.9.1
• v0.9.2
• v0.9.3
• v0.9.4
• v0.9.6
• v2-endpoints
• vue
• vue-explorer
• vue-integration
• web-edit-meta
• web-list-metas

Build # 931

Build Type

Pull #345

travis-ci

Committed by web-flow

Commit Message

Ingest Bug Fix

We were having an issue consuming a few data sets. After some
investigation, I discovered that it was due to quotes being used in
header rows:

```
One,Two,"Three, or three",Four
```

This would then be translated into the Copy SQL as:

```
COPY ... ("One", "Two", ""Three, or three"", "Four") ...
```

which obviously isn't going to work.

The solution is to _regex_ split the header line rather than the naive
_string_ split. But you may ask, "Why do splits at all? Can't we just
use the CSV module and pull the keys from the first map row?". The
unfortunate answer to this is that the CSV module does not preserve
order -- if you have a header row `a,c,b` then you will get a map with
keys ordered as `a,b,c`. To preserve the order of the headers, you need
to split the line.

Fixes #344

Pull Request Pull Request #345: Ingest Bug Fix

Run Details

12 of 12 new or added lines in 3 files covered. (100.0%)

1368 of 1803 relevant lines covered (75.87%)

65.74 hits per line