Kapiche / caterpillar

100%

master: 100%

LAST BUILD ON BRANCH issue_36

branch: issue_36

CHANGE BRANCH

x

Reset

• issue_36
• 0.10.0
• 0.11.0
• 0.12.0
• 0.13.0
• 0.14.0
• 0.14.2
• 0.15.0
• 0.15.1
• 0.15.10
• 0.15.2
• 0.15.3
• 0.15.4
• 0.15.5
• 0.15.6
• 0.15.7
• 0.15.8
• 0.15.9
• 1.0.0.dev11
• 1.0.0.dev12
• 1.0.0.dev15
• 1.0.0.dev16
• 1.0.0.dev17
• 1.0.0.dev6
• 1.0.0.dev7
• 1.0.0.dev8
• 1.0.1.dev15
• Bigrams-Gen
• PROD-213
• WIP_serialisation_format
• chrysalis
• cleanup_api_pagination
• co-occurence
• date-search
• explicit-encoding-readme
• filter_non_empty
• fix-ngram-search
• fix_attribute_field_selection
• flake8-fixes
• frame_attribute_api
• frame_metadata
• handle_unicode_surrogates
• initialise_migration_framework
• issue_11
• issue_18
• issue_21
• issue_41
• issue_47
• issue_54
• master
• metadata-searching-fix
• metadata-segments
• non_destructive_indexing
• not-in-search
• per_field_index
• plugin-transactions
• primitive_search_documents
• python3
• query-fixes
• refactor_plugin
• revert-144-cjrh-py37
• sanitise_field_names
• scoring_and_ngrams
• serialisation_format
• serialisation_format_experiment
• skipgrams
• stopword_contractions
• stopwords
• store_all_positions
• term_positions
• tfidf_improvements
• tokenisation
• transactional_schema_config
• update_test_runner
• upper_pagination_key

Build # 52

Build Type

push

travis-ci

Committed by rstuart85

Commit Message

New whitespace word tokenizer and punctuation filter

Tokenisation of words now defaults to a tokeniser that splits on
whitespace and captures compound names. The default analyser now also
uses a filter that allows certain leading and trailing punctuation and a
filter that removes possessive contractions in a really robust manner.

There were a bunch of changes to test that were really annoying. I've
now created test analyers that should change ever. When new tokenisers
and analysers are added, we should only ahve to add tests to the
test_analyse etc., not adjust all tests. Of the tests that were updated:

* test_index_merge_terms: An instance of party became tea-party
* test_index_moby_case_folding: Multiple cases of terms being joined to
* other terms because of punctuation. For example - 'Flask--good-bye'
* test_searching_reserved_words: Multiple cases of 'and' joined to other
* terms with punctuation
* test_index_alice_bigram_discovery: The two missing bigrams ('beau
  ootiful' and 'guinea pigs') and now single terms joined with 1
* or more '-'.
* test_searching_alice: Two places where a *ing ending term was joined
* to another term with 1 or more '-'.
* test_searching_alice: One case of 'voice' term joined with '-'.
* test_index_alice: All differences in vocab size verified by hand.

Also fixed a small bug with setup.py.

Coverage Stats

2791 of 2791 relevant lines covered (100.0%)

1.0 hits per line