PyThaiNLP / pythainlp

96%

dev: 52%

LAST BUILD ON BRANCH fixed-#609

branch: fixed-#609

CHANGE BRANCH

x

Reset

• fixed-#609
• 2.1
• 2.2
• 2.3
• 3.0
• 3.1
• 3.1.1-releases
• 4.0
• 4.0-beta1
• 4.1.0-beta1
• 5.1
• 5.1-patch
• Add-CITATION
• Add-new-thaiwords
• Add-provinces
• Add-th-fr-mt
• Add-th-zh-translate
• Add-thai-docs
• Add-thai-text-strptime
• Fix-666
• Fixed-#644
• Fixed-#645
• add-1075
• add-LST20-postag
• add-Thai-English-soundex
• add-abbreviation_to_full_text
• add-ancient
• add-best-corpus-reader
• add-blackboard-cls
• add-blackboard-postag
• add-budoux
• add-calculate_ngram_counts
• add-clause
• add-clean-maiyamok
• add-convert_currency
• add-copyright
• add-coref
• add-correct-engine
• add-count_thai_chars
• add-create_wordlist
• add-dependency-parser
• add-deprecated-word_vector
• add-dict-input-norvigspell
• add-editorconfig
• add-el
• add-encoding
• add-family-names
• add-find_synonym
• add-geo_neo
• add-han_solo
• add-hf-hub
• add-iso_11940-transliteration
• add-khavee-docs-tests
• add-live-dead-syllable-classify
• add-lst20-ner-onnx
• add-ltw2v-word_vector
• add-more-g2p
• add-ner-class
• add-nercut
• add-new-misspelling
• add-new-science-word
• add-nlpo3
• add-orst-list-and-change-correct
• add-perceptron-tagger
• add-phoneme-util
• add-pipline
• add-pythainlp-chat
• add-pythainlp.morpheme
• add-remove_tone_ipa
• add-retrained-crfcut-model
• add-rhyme
• add-save-load-param_free
• add-small100
• add-spdx-filetype
• add-spell_words
• add-spelling
• add-sponsors
• add-spoonerism
• add-syllable_tokenize
• add-tcc-test
• add-text-generator
• add-textaugment
• add-th_tdtb
• add-thai-lunar-date
• add-thai-morse
• add-thai-number2arabic-digit
• add-thai_nner
• add-thai_pangram
• add-thai_synonym
• add-thai_to_idn
• add-thai_w2p_v0.2
• add-thaig2p_v2
• add-thainer-model
• add-thainer-v2
• add-tltk
• add-tone-detector
• add-translate
• add-translate-class
• add-tud
• add-ud_goeswith
• add-umlfit-test
• add-update-dict
• add-windows-macos-ci
• add-word-dict
• add-word2word
• add-word_approximation
• add-word_detokenize
• add-wsd
• add-wtpsplit
• add-wunsen
• benchmark-to-cli
• better-readme
• change-corpus-license
• change-coverage2github
• corpus-version
• delete-lst20-model
• dependabot/github_actions/actions/checkout-5
• dependabot/github_actions/actions/checkout-6
• dependabot/github_actions/actions/download-artifact-4
• dependabot/github_actions/actions/download-artifact-5
• dependabot/github_actions/actions/download-artifact-6
• dependabot/github_actions/actions/first-interaction-2
• dependabot/github_actions/actions/first-interaction-3
• dependabot/github_actions/actions/upload-artifact-4
• dependabot/github_actions/actions/upload-artifact-5
• dependabot/github_actions/astral-sh/ruff-action-2
• dependabot/github_actions/astral-sh/ruff-action-3
• dependabot/github_actions/github/codeql-action-4
• dependabot/pip/bpemb-0.3.6
• dependabot/pip/emoji-0.6.0
• dependabot/pip/emoji-1.7.0
• dependabot/pip/emoji-2.14.0
• dependabot/pip/epitran-1.25.1
• dependabot/pip/epitran-1.26.0
• dependabot/pip/epitran-1.34.0
• dependabot/pip/esupar-1.6.9
• dependabot/pip/esupar-1.7.5
• dependabot/pip/fairseq-0.12.2
• dependabot/pip/fastai-2.6.3
• dependabot/pip/fastai-2.7.18
• dependabot/pip/gensim-4.3.3
• dependabot/pip/h5py-3.12.1
• dependabot/pip/h5py-3.13.0
• dependabot/pip/khanaa-0.1.1
• dependabot/pip/nltk-3.6.5
• dependabot/pip/numpy-eq-1.26.star
• dependabot/pip/numpy-eq-2.0.star
• dependabot/pip/numpy-eq-2.1.star
• dependabot/pip/numpy-eq-2.2.star
• dependabot/pip/numpy-gte-1.26.0-and-lt-3
• dependabot/pip/pandas-eq-2.2.star
• dependabot/pip/panphon-0.21.2
• dependabot/pip/panphon-0.22.2
• dependabot/pip/protobuf-3.20.2
• dependabot/pip/protobuf-5.29.1
• dependabot/pip/protobuf-5.29.2
• dependabot/pip/protobuf-5.29.3
• dependabot/pip/protobuf-6.30.1
• dependabot/pip/pyicu-2.14
• dependabot/pip/pyicu-2.15
• dependabot/pip/pyicu-2.15.1
• dependabot/pip/pyicu-2.15.2
• dependabot/pip/pyyaml-6.0.2
• dependabot/pip/requests-eq-2.32.star
• dependabot/pip/sacremoses-0.1.1
• dependabot/pip/sentence-transformers-2.7.0
• dependabot/pip/sentence-transformers-3.2.1
• dependabot/pip/sentence-transformers-3.3.1
• dependabot/pip/sentence-transformers-4.0.1
• dependabot/pip/sentence-transformers-4.0.2
• dependabot/pip/sentence-transformers-4.1.0
• dependabot/pip/sentence-transformers-gte-2.7.0-and-lt-6
• dependabot/pip/sentencepiece-0.2.0
• dependabot/pip/spacy-eq-3.8.star
• dependabot/pip/spacy-thai-0.7.8
• dependabot/pip/spylls-0.1.7
• dependabot/pip/symspellpy-6.7.8
• dependabot/pip/symspellpy-6.9.0
• dependabot/pip/tensorflow-2.11.1
• dependabot/pip/tensorflow-2.18.0
• dependabot/pip/tensorflow-2.18.1
• dependabot/pip/tensorflow-2.4.2
• dependabot/pip/tensorflow-2.5.1
• dependabot/pip/tensorflow-2.5.2
• dependabot/pip/tensorflow-2.5.3
• dependabot/pip/tensorflow-2.6.4
• dependabot/pip/tensorflow-2.9.3
• dependabot/pip/torch-1.13.1
• dependabot/pip/torch-2.4.1
• dependabot/pip/torch-2.5.1
• dependabot/pip/transformers-4.30.0
• dependabot/pip/transformers-4.36.0
• dependabot/pip/transformers-4.38.0
• dependabot/pip/transformers-4.47.0
• dependabot/pip/transformers-4.47.1
• dependabot/pip/transformers-4.48.0
• dependabot/pip/transformers-4.48.1
• dependabot/pip/transformers-4.48.2
• dependabot/pip/transformers-4.48.3
• dependabot/pip/transformers-4.49.0
• dependabot/pip/transformers-4.50.3
• dependabot/pip/transformers-4.51.0
• dependabot/pip/transformers-4.51.3
• dependabot/pip/transformers-4.52.3
• dependabot/pip/transformers-4.53.0
• dependabot/pip/transformers-4.53.1
• dependabot/pip/transformers-4.53.2
• dependabot/pip/transformers-4.54.1
• dependabot/pip/transformers-4.55.0
• dependabot/pip/transformers-4.55.2
• dependabot/pip/transformers-4.55.4
• dependabot/pip/transformers-4.57.0
• dependabot/pip/transformers-4.57.1
• dependabot/pip/transformers-4.57.3
• dependabot/pip/ufal-chu-liu-edmonds-1.0.3
• dependabot/pip/wtpsplit-1.3.0
• dependabot/pip/wtpsplit-2.1.1
• deprecate-conceptnet
• dev
• display-thai-char
• fix-1004
• fix-461
• fix-ci-bug
• fix-corpus-utf8
• fix-deprecate-word-vec
• fix-dockerfile
• fix-etcc-dict
• fix-gensim
• fix-iso-11940-dup-keys
• fix-karan-tcc
• fix-maiyamok
• fix-symspellpy
• fix-test
• fix-thai_strftime
• fix-tokenization-benchmark
• fix-umlfit
• fix-update-bug
• fix/help
• fix_docs_ci
• fixed-1055
• hotfix-bug-normalize
• improve-1
• korakot-patch-1
• llm-tool
• merge-syllable-subword
• move-model
• patch-1110
• pythainlp-3.0
• pyup-config
• remotes/origin/add-family-names
• remotes/origin/korakot-patch-1
• remove-clause_tokenize
• remove-code-encodings
• remove-conceptnet
• remove-deprecated-function
• remove-spylls
• retrain-ud-pud-postag
• simpify-calculate_f_year_f_dev
• start-2.4
• start-3.0
• start-beta1
• text-cleaning
• thai-ner-1-5-2
• thai-readme
• unittest-main
• update-actions-version
• update-attacut
• update-docker-fairseq
• update-docs
• update-import
• update-import-2.2
• update-import-python-crfsuite
• update-maiyamok
• update-nlpo3-python-v1.2.2
• update-postag-docs
• update-pythainlp-download-corpus
• update-royin
• v2.3.2
• v2.4.0-dev0
• v3.0.0
• v3.0.0-beta
• v3.0.0-dev0
• v3.0.1
• v3.0.10
• v3.0.2
• v3.0.3
• v3.0.5
• v3.0.6
• v3.0.9
• v3.1.0
• v3.1.0-beta0
• v3.1.0-dev0
• v3.1.0-dev1
• v3.1.0-dev2
• v3.1.0-dev3
• v3.1.1
• v4.0.0
• v4.0.0-beta1
• v4.0.2
• v4.1.0-beta1
• v4.1.0-beta2
• v4.1.0-beta3
• v4.1.0-beta5
• v5.0.0
• v5.0.0-beta1
• v5.0.0-dev0
• v5.0.0-dev1
• v5.0.0-dev2
• v5.0.1
• v5.0.2
• v5.0.3
• v5.0.4
• v5.1.0
• v5.1.0-beta1
• v5.1.1
• v5.1.2
• wannaphong/add-codemeta
• wannaphong/add-display-cell-tokenizer
• wannaphong/add-lcs
• wannaphong/add-misspell-command
• wannaphong/fix-custom-dict-error
• wannaphong/fix-newmm-safe
• wannaphong/fix-tokenizer

Update test_tokenize.py