MITLibraries / timdex-semantic-builder / 22781969752

97%

main: 89%

push

github

Adds tokenizer lambda

Why are these changes being introduced:

* This is the core work to put the tokenizer lambda in place, which is a
  critical component of the overall system. It will be used to tokenize
  queries for use in the search process.

Relevant ticket(s):

* https://mitlibraries.atlassian.net/browse/USE-427

How does this address that need:

* This change introduces the tokenizer lambda
* It includes the code for the lambda itself, as well as tests and
  documentation for how to use it.

Document any side effects to this change:

* Adjusted some linting rules from the template to allow better
  understanding of what we are skipping and added some skips as
  appropriate (mostly around tests).
* leaves templated `my_function` in place as a placeholder. This will
  likely be removed in the near future.
* Introduces files from a Huggingface model without an automated way
  to update them. Whether this is temporary or not will be based on how
  often we need to update those files and whether we can find a better
  way to manage them.

60 of 60 new or added lines in 2 files covered. (100.0%)

120 of 122 relevant lines covered (98.36%)

0.98 hits per line