mysociety / pombola / 3450 / 1
60%
master: 60%

DEFAULT BRANCH: master
Ran
Files 390
Run time 33s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 54.513% (+0.006%) from 54.507%
ES_VERSION=0.90.13 ES_DOWNLOAD_URL=https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-${ES_VERSION}.tar.gz

push

travis-ci

jacksonj04 
Add fuzzy search operator to searches.

Haystack prevents us from easily applying a [fuzziness to the whole search](https://www.elastic.co/guide/en/elasticsearch/reference/0.90/query-dsl-fuzzy-query.html),
so this is a workaround which checks to see if a string is entirely
alphanumeric ASCII characters and spaces, then applies the fuzzy query
string operator to individual words if so. If not, `AutoQuery()` handles
escaping special characters and turning boolean patterns such as `AND` into
actual structured query objects.

For future reference, this workaround is necessary without radically re-working search throughout Pombola to use less Haystack magic and allow us to modify search kwargs as [suggested on Stack Overflow](https://stackoverflow.com/questions/18000714/how-can-i-do-a-fuzzy-search-using-django-haystack-and-the-elasticsearch-backend). @mhl and I spent quite a bit of time digging through exactly what was going on, and this appears to be the most logical solution without serious work.

This does have the following caveats:

* Anything which deviates from a boring alphanumeric ASCII query will not be fuzzed at all.
* Trying to include boolean patterns via keywords (`OR`) rather than operators (`|`) will break, since the keywords will also be fuzzed.
* We have no way to adjust any value other than the edit distance (currently set at 1) using query string operators.

For anybody stumbling this way in future, we discovered:

* `build_search_kwargs()` as part of an extended `ElasticsearchSearchBackend` class is definitely where you _would_ logically tell ElasticSearch to make a search fuzzy, however;
* `AutoQuery()` turns a string into a Haystack structured query object, which when it gets passed to `build_search_kwargs()` makes it nigh-on impossible to extract the original term to use as a value in providing a fuzzy argument, and;
* Haystack, in `elasticsearch_backend.py` goes back and stomps over your kwargs in unexpected ways.

Recommendations for the future are:

* Move away from ElasticSearch 0.90 at an opportune moment (probably an architecture refresh, this is quite a lot of work to do purely for its own sake), and;
* Use that opportunity to overhaul searching in Pombola, and move from Haystack to [elasticsearch-dsl](https://elasticsearch-dsl.readthedocs.io/en/latest/).

10280 of 18858 relevant lines covered (54.51%)

0.55 hits per line

Source Files on job 3450.1 (ES_VERSION=0.90.13 ES_DOWNLOAD_URL=https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-${ES_VERSION}.tar.gz)