PyThaiNLP / pythainlp

96%

dev: 53%

LAST BUILD ON BRANCH fixed-#609

branch: fixed-#609

CHANGE BRANCH

x

Reset

• fixed-#609
• dev
• 2.1
• fix-tokenization-benchmark
• change-coverage2github
• unittest-main
• add-umlfit-test
• corpus-version
• benchmark-to-cli
• add-word-dict
• add-translate
• add-dict-input-norvigspell
• 2.2
• fix-thai_strftime
• fix-update-bug
• add-best-corpus-reader
• add-sponsors
• change-corpus-license
• fix_docs_ci
• fix-etcc-dict
• better-readme
• thai-readme
• Add-provinces
• display-thai-char
• add-LST20-postag
• add-perceptron-tagger
• add-family-names
• remotes/origin/add-family-names
• update-postag-docs
• add-clause
• hotfix-bug-normalize
• korakot-patch-1
• remotes/origin/korakot-patch-1
• update-attacut
• fix-test
• add-windows-macos-ci
• add-create_wordlist
• add-nercut
• update-import
• update-import-2.2
• fix-gensim
• 2.3
• remove-code-encodings
• merge-syllable-subword
• update-royin
• fix-461
• fix-deprecate-word-vec
• add-thainer-model
• add-textaugment
• add-ltw2v-word_vector
• add-geo_neo
• add-thai-number2arabic-digit
• Add-th-zh-translate
• add-translate-class
• start-2.4
• add-correct-engine
• add-text-generator
• add-retrained-crfcut-model
• add-deprecated-word_vector
• dependabot/pip/tensorflow-2.4.2
• add-tltk
• add-ner-class
• retrain-ud-pud-postag
• v2.4.0-dev0
• start-3.0
• v2.3.2
• dependabot/pip/tensorflow-2.5.1
• dependabot/pip/nltk-3.6.5
• pyup-config
• add-nlpo3
• add-clean-maiyamok
• add-spoonerism
• dependabot/pip/tensorflow-2.5.2
• update-import-python-crfsuite
• update-nlpo3-python-v1.2.2
• deprecate-conceptnet
• Add-new-thaiwords
• v3.0.0-dev0
• Add-th-fr-mt
• v3.0.0-beta
• v3.0.0
• Fixed-#645
• Fixed-#644
• v3.0.1
• v3.0.2
• v3.0.3
• dependabot/pip/tensorflow-2.5.3
• v3.0.5
• add-update-dict
• 3.0
• update-pythainlp-download-corpus
• add-lst20-ner-onnx
• add-iso_11940-transliteration
• add-thai_w2p_v0.2
• Add-thai-docs
• add-thai_nner
• Fix-666
• v3.0.6
• dependabot/pip/tensorflow-2.6.4
• add-live-dead-syllable-classify
• fix-ci-bug
• add-wunsen
• remove-deprecated-function
• add-tone-detector
• 3.1
• v3.1.0-dev0
• v3.1.0-dev1
• add-word_detokenize
• add-Thai-English-soundex
• fix-corpus-utf8
• pythainlp-3.0
• v3.0.9
• v3.1.0-dev2
• move-model
• add-dependency-parser
• thai-ner-1-5-2
• v3.1.0-dev3
• v3.1.0-beta0
• v3.0.10
• v3.1.0
• improve-1
• Add-CITATION
• delete-lst20-model
• add-blackboard-cls
• add-blackboard-postag
• fix-umlfit
• fix-karan-tcc
• add-tcc-test
• add-count_thai_chars
• 3.1.1-releases
• v3.1.1
• add-ud_goeswith
• dependabot/pip/tensorflow-2.9.3
• add-pipline
• add-new-science-word
• Add-thai-text-strptime
• dependabot/pip/torch-1.13.1
• add-phoneme-util
• add-remove_tone_ipa
• add-khavee-docs-tests
• add-thainer-v2
• add-copyright
• dependabot/pip/tensorflow-2.11.1
• 4.0-beta1
• v4.0.0-beta1
• 4.0
• v4.0.0
• v4.0.2
• add-coref
• add-wtpsplit
• dependabot/pip/transformers-4.30.0
• add-word_approximation
• add-orst-list-and-change-correct
• add-encoding
• add-small100
• add-spell_words
• add-pythainlp-chat
• add-wsd
• add-el
• add-thai_synonym
• add-abbreviation_to_full_text
• 4.1.0-beta1
• v4.1.0-beta1
• v4.1.0-beta2
• add-han_solo
• v4.1.0-beta3
• add-ancient
• add-syllable_tokenize
• add-new-misspelling
• v4.1.0-beta5
• add-rhyme
• fix-iso-11940-dup-keys
• v5.0.0-dev0
• add-thai_to_idn
• dependabot/pip/protobuf-3.20.2
• add-find_synonym
• add-thai-morse
• v5.0.0-dev1
• dependabot/pip/transformers-4.36.0
• add-pythainlp.morpheme
• v5.0.0-dev2
• v5.0.0-beta1
• v5.0.0
• v5.0.1
• v5.0.2
• dependabot/pip/transformers-4.38.0
• add-save-load-param_free
• add-editorconfig
• add-th_tdtb
• v5.0.3
• add-tud
• fix-maiyamok
• v5.0.4
• add-thaig2p_v2
• dependabot/pip/sentencepiece-0.2.0
• dependabot/pip/esupar-1.7.5
• dependabot/pip/fastai-2.7.18
• dependabot/pip/symspellpy-6.7.8
• dependabot/pip/numpy-eq-2.1.star
• dependabot/pip/tensorflow-2.18.0
• dependabot/pip/bpemb-0.3.6
• dependabot/pip/esupar-1.6.9
• dependabot/pip/h5py-3.12.1
• dependabot/pip/spacy-eq-3.8.star
• dependabot/github_actions/actions/upload-artifact-4
• dependabot/github_actions/actions/download-artifact-4
• add-thai-lunar-date
• dependabot/pip/numpy-eq-2.0.star
• dependabot/pip/torch-2.5.1
• dependabot/pip/pyyaml-6.0.2
• dependabot/pip/sentence-transformers-3.3.1
• dependabot/pip/requests-eq-2.32.star

Update test_tokenize.py