microsoft / teams-ai / 8084317865 / 1 – python
80%
main: 80%

DEFAULT BRANCH: main
Ran
Files 111
Run time 7s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 89.444%. Remained the same
python – 8084317865.1

push

github

web-flow 
[JS] fix: Change default encoder for Tokenizer (#1352)

## Linked issues

closes: #1171 

## Details

In favor of more accurate token calculation, I have removed
`gpt-3-encoder` and replaced with `gpt-tokenizer`, which is a TypeScript
rewrite and modernization of the original package. By default,
`gpt-tokenizer` uses `cl100k_base` (gpt 3.5 and gpt 4) encoding and can
be configured as needed to use different encoders for other models.

As an example, GPT3Tokenizer has been updated to use encoder/decoder for
text-davinci-003 and can be passed in to be used in LLMClient. (See
`GPT3Tokenizer`)

- Remove package `gpt-3-encoder`
- Add package `gpt-tokenizer` (to replace above)
- Create `GPTTokenizer.ts` as the default encoder.
- `GPT3Tokenizer` now uses `gpt-tokenizer`
- Add unit tests / fix unit tests to use `GPTTokenizer`


## Other changes
- Update docs (`teams-ai.api.json` and `teams-ai.api.md`)
- Prevent test files from being published (`tsconfig`)
- Update inline docs to Application, Meetings, Utilities,
ActionAugmentationSection, DefaultAugmentation, MonologueAugmentation,
SequenceAugmentation, TeamsBotSSOPrompt,
TeamsSSOAdaptiveCardAuthentication, TeamsSSOBotAuthentication,
TeamsSSOMessageExtensionAuthentication, TextDataSource, TestMemoryFork,
OpenAIModel, ActionPlanner, AssistantsPlanner, LLMClient, Prompt,
PromptManager,
- Turn off "no-prototype-builtins" eslint rule
- Fix broken link in `/js/packages/teams-ai/README.md`

## Attestation Checklist

- [x] My code follows the style guidelines of this project

- I have checked for/fixed spelling, linting, and other errors
- I have commented my code for clarity
- I have made corresponding changes to the documentation (updating the
doc strings in the code is sufficient)
- My changes generate no new warnings
- I have added tests that validates my changes, and provides sufficient
test coverage. I have tested with:
  - Local testing
  - E2E testing in Teams... (continued)

3135 of 3505 relevant lines covered (89.44%)

0.89 hits per line

Source Files on job python - 8084317865.1