microsoft / teams-ai / 8084317865

80%

push

github

[JS] fix: Change default encoder for Tokenizer (#1352)

## Linked issues

closes: #1171 

## Details

In favor of more accurate token calculation, I have removed
`gpt-3-encoder` and replaced with `gpt-tokenizer`, which is a TypeScript
rewrite and modernization of the original package. By default,
`gpt-tokenizer` uses `cl100k_base` (gpt 3.5 and gpt 4) encoding and can
be configured as needed to use different encoders for other models.

As an example, GPT3Tokenizer has been updated to use encoder/decoder for
text-davinci-003 and can be passed in to be used in LLMClient. (See
`GPT3Tokenizer`)

- Remove package `gpt-3-encoder`
- Add package `gpt-tokenizer` (to replace above)
- Create `GPTTokenizer.ts` as the default encoder.
- `GPT3Tokenizer` now uses `gpt-tokenizer`
- Add unit tests / fix unit tests to use `GPTTokenizer`


## Other changes
- Update docs (`teams-ai.api.json` and `teams-ai.api.md`)
- Prevent test files from being published (`tsconfig`)
- Update inline docs to Application, Meetings, Utilities,
ActionAugmentationSection, DefaultAugmentation, MonologueAugmentation,
SequenceAugmentation, TeamsBotSSOPrompt,
TeamsSSOAdaptiveCardAuthentication, TeamsSSOBotAuthentication,
TeamsSSOMessageExtensionAuthentication, TextDataSource, TestMemoryFork,
OpenAIModel, ActionPlanner, AssistantsPlanner, LLMClient, Prompt,
PromptManager,
- Turn off "no-prototype-builtins" eslint rule
- Fix broken link in `/js/packages/teams-ai/README.md`

## Attestation Checklist

- [x] My code follows the style guidelines of this project

- I have checked for/fixed spelling, linting, and other errors
- I have commented my code for clarity
- I have made corresponding changes to the documentation (updating the
doc strings in the code is sufficient)
- My changes generate no new warnings
- I have added tests that validates my changes, and provides sufficient
test coverage. I have tested with:
  - Local testing
  - E2E testing in Teams... (continued)

2632 of 3775 branches covered (69.72%)

Branch coverage included in aggregate %.

9409 of 11181 relevant lines covered (84.15%)

18.29 hits per line