Kapiche / caterpillar

100%

master: 100%

LAST BUILD ON BRANCH handle_unicode_surrogates

branch: handle_unicode_surrogates

CHANGE BRANCH

x

Reset

• handle_unicode_surrogates
• 0.10.0
• 0.11.0
• 0.12.0
• 0.13.0
• 0.14.0
• 0.14.2
• 0.15.0
• 0.15.1
• 0.15.10
• 0.15.2
• 0.15.3
• 0.15.4
• 0.15.5
• 0.15.6
• 0.15.7
• 0.15.8
• 0.15.9
• 1.0.0.dev11
• 1.0.0.dev12
• 1.0.0.dev15
• 1.0.0.dev16
• 1.0.0.dev17
• 1.0.0.dev6
• 1.0.0.dev7
• 1.0.0.dev8
• 1.0.1.dev15
• Bigrams-Gen
• PROD-213
• WIP_serialisation_format
• chrysalis
• cleanup_api_pagination
• co-occurence
• date-search
• explicit-encoding-readme
• filter_non_empty
• fix-ngram-search
• fix_attribute_field_selection
• flake8-fixes
• frame_attribute_api
• frame_metadata
• initialise_migration_framework
• issue_11
• issue_18
• issue_21
• issue_36
• issue_41
• issue_47
• issue_54
• master
• metadata-searching-fix
• metadata-segments
• non_destructive_indexing
• not-in-search
• per_field_index
• plugin-transactions
• primitive_search_documents
• python3
• query-fixes
• refactor_plugin
• revert-144-cjrh-py37
• sanitise_field_names
• scoring_and_ngrams
• serialisation_format
• serialisation_format_experiment
• skipgrams
• stopword_contractions
• stopwords
• store_all_positions
• term_positions
• tfidf_improvements
• tokenisation
• transactional_schema_config
• update_test_runner
• upper_pagination_key

Build # 318

Build Type

push

travis-ci

Committed by SamHames

Commit Message

Switch from ujson to standard library json

ujson is severely limited in how it handles a variety of unicode
problems, particularly stray utf16 surrogates in otherwise well
encoded text. In particular, ujson.loads() can generate strings
that cannot work with ujson.dumps().

The standard library json module allows the stray surrogates
through without further processing. Because the source data is
already broken it seems best we don't attempt to make further
modification and pass it through the chain as best we can.

Coverage Stats

3080 of 3080 relevant lines covered (100.0%)

2.0 hits per line