ElixirMentor / prompt_vault / 81fdc262c53e1cc01f5c0d1966562ba1ff8f0219-PR-2

84%

main: 84%

Pull #2

github

Add TiktokenTokenizer for accurate OpenAI token counting

- Introduced TiktokenTokenizer to provide precise token counts for OpenAI models, including GPT-4 and GPT-3.5-turbo.
- Updated mix.exs to include the tiktoken dependency.
- Enhanced README.md with installation instructions and usage examples for TiktokenTokenizer.
- Added comprehensive tests for TiktokenTokenizer functionality.
- Created CHANGELOG.md to document notable changes and updates.

These changes improve token counting accuracy and enhance the overall functionality of the PromptVault library.

10 of 11 new or added lines in 1 file covered. (90.91%)

151 of 180 relevant lines covered (83.89%)

7.11 hits per line