OpenTreeMap / otm-core / 2952
84%
develop: 84%

LAST BUILD BRANCH: dependabot/npm_and_yarn/sockjs-0.3.21
DEFAULT BRANCH: develop
Ran
Jobs 1
Files 216
Run time 11s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
2952

push

travis-ci

RobinIsTheBird 
upgrade HStore

Overview
--------

The point of this commit is to switch from
`django_hstore.fields.DictionaryField` to
`django.contrib.postgres.fields.HStoreField`.

The former is incompatible with later releases of django,
and OpenTreeMap needs to migrate from the current 1.8 to 1.11
for stability.

The latter is the sanctioned django interface to PostgreSQL's hstore
going forward.

The latter is also less functional than the former, prompting a series
of other changes.
Changes were also made for the sake of maintenance simplicity.

The key changes here are in `treemap.udf`, `treemap.audit`, `*.models`, and
`treemap.search`.

**treemap.udf**

Entry points:
-   `UDFModel`, a base class for models that need user defined fields
-   `UserDefinedFieldDefinition`, defines the name, data type, and model type
    for a user defined field, for a given treemap instance, as a
    scalar or a collection
-   `UserDefinedCollectionValue`, an HStore collection value for a
    specific `UserDefinedFieldDefinition` on a specific model instance

`UDFModel` user guide:

Classes that want custom fields subclass `UDFModel`.
The `UDFModel` `HStoreField` is named `hstore_udfs`, but unless you are
getting deep into udf code, you should use `udfs`, which is an ordinary
attribute on the model instance, not directly associated with any
django `Field`.

To assign a custom field,
    `my_model_instance.udfs['custom field name'] = value`

There are three ways to retrieve it:
    `my_model_instance.udfs['custom field name']`
    `getattr(my_model_instance, 'custom field name')`
    `getattr(my_model_instance, 'udf:custom field name'`)

All of the above work for both scalar and collection custom fields.

Filter using the `hstore_udf` transform.

In addition to the transforms and lookups described at
https://docs.djangoproject.com/en/1.8/ref/contrib/postgres/fields/,
`UDFModel` implements `__int` and `__float` transforms, for use
before magnitude comparisons (`__gt`, `__gte`, `__lt`, `__lte`).

`udfs` perpetually syncs to `hstore_udfs`.
Values in `udfs` are always typed for the app, and
values in `hstore_udfs` are always typed for the db.
Since they are perpetually clean, code that assigns to `udfs`
needs to be prepared for a `ValidationError` at the time of the
assignment, not just at `save`.

In practice, this is not a problem, because the web frontend
takes care to send valid data in PUTs and POSTs.

**treemap.audit and \*.models**

- Models can use a plain `GeoManager` now
- In order to reduce code coupling and assure that initialization happens in a
  rational order, `populate_previous_state` is now the responsibility of the
  leaf subclass of `UserTrackable`.
- Species is no longer a `UDFModel` because it never employed udfs,
  and removing unused code makes maintenance simpler.

Before this change, `populate_previous_state` was getting called between
initialization of different levels of model class. It calls `as_dict`, which
raises exceptions if not all fields have been initialized,
so now `populate_previous_state` is the responsibility of the leaf subclass.

This reduces the coupling between auditing and custom fields.

That tight coupling was why previous versions of `udfs.py` had to
inject a descriptor into the django model instantiation procedure.
Descriptors are fine as long as they work, but when they don't, they
are a headache to debug. Better to reduce the amount of injection
into django's procedure.

**treemap.search**

- Use the new `hstore_udf` filter transform
- Additional parser steps to add the udf `__float` filter transform

Specific Changes
----------------
- Replace UDF specific GeoHStoreUDFManager with generic
  GeoManager in treemap and stormwater models
- Add django.contrib.postgres to settings
- Migration to give the `udfs` column a default
- Migration to demote Species from UDFModel
- `treemap.udf`
  - Eliminate
    - UDF specific manager and descriptor
    - `quotesingle`
    - `UDF_ORDER_PATTERN`
    - `UDFBaseContains(Lookup)`
    - `UDFContains` and `UDFIcontains`
    - `UDFQuery(Query)`
    - `UDFQuerySet(HStoreGeoQuerySet)`
    - `GeoHStoreUDFManager(HStoreGeoManager)`
    - unused `'user'` datatype
  - Alias `UDFField` to `HStoreField` for the sake of earlier migrations
  - Rename the `HStoreField` to `hstore_udfs` with a column param
    to point it at the `udfs` column in `treemap_mapfeature` table
  - Dynamically make a `udfs` model instance attribute in `UDFModel.__init__`
  - Migration to support the above
    - The auto-generated migration used `RemoveField` and `AddField`,
      which lost all the data so the migration here has been hacked to use
      `RenameField` and `AlterField`, and tested for data retention.
  - `UserDefinedCollectionValue.data` now based on
    `django.contrib.postgres.fields.HStoreField`
  - If a collection field choice is removed,
    delete all UDCVs that name that choice,
    not just the choice field in the UDCV
  - Use `feature_type` attribute if it exists, because the only classname
    available might be the generic `MapFeature`.
  - Have a stackoverflow reference in case ordering
    by an hstore key ever becomes important
  - Perhaps it's a design flaw that `udfs` perpetually cleans and
    `hstore_udfs` perpetually reverse cleans, and both should be
    delayed to `save_with_user`, but there are disadvantages to
    allowing dirty values, too.
  - Transforms based on
    `django.contrib.postgres.fields.hstore.KeyTransform`,
    which is registered on `HStoreField`
- Search now works for both collection and scalar UDFs
- `treemap.search`
  - Removed 'IN' and 'ISNULL' predicates from collection search
    because they are never used in production
  - Removed 'WITHIN_RADIUS' predicate from scalar search
    because it is never used in production
- Removed tests that no longer apply, or that test code that
  is never used in production
- Modified integer and float udf tests so that they fail if the emitted
  SQL does a lexical comparison to demonstrate the problem
- Add a `'multichoice'` test to `test_udfs`
- Fix form_extras to tolerate a missing UDF key
- Test for `is None` wherever a falsey value could be legitimate
- Adapt `treemap.views.map_feature.update_map_feature` to include
  `ValidationError` `message_dict`s from setting a udf to an invalid value
  in its `errors`.

Testing Notes
-------------
- Combines well with other searches
- ... unless you are bitten by #3080

--

Connects to #1968

17181 of 20596 relevant lines covered (83.42%)

0.83 hits per line

Jobs
ID Job ID Ran Files Coverage
1 2952.1 0
83.42
 Travis Job 2952.1
Source Files on build 2952
Detailed source file information is not available for this build.