PyThaiNLP / pythainlp

54%

dev: 67%

LAST BUILD ON BRANCH copilot/fix-text-normalization-issue

branch: copilot/fix-text-normalization-issue

CHANGE BRANCH

x

Reset

• copilot/fix-text-normalization-issue
• 2.1
• 2.2
• 2.3
• 3.0
• 3.1
• 3.1.1-releases
• 4.0
• 4.0-beta1
• 4.1.0-beta1
• 5.1
• 5.1-patch
• Add-CITATION
• Add-new-thaiwords
• Add-provinces
• Add-th-fr-mt
• Add-th-zh-translate
• Add-thai-docs
• Add-thai-text-strptime
• Fix-666
• Fixed-#644
• Fixed-#645
• add-1075
• add-LST20-postag
• add-Thai-English-soundex
• add-abbreviation_to_full_text
• add-ancient
• add-best-corpus-reader
• add-blackboard-cls
• add-blackboard-postag
• add-budoux
• add-calculate_ngram_counts
• add-clause
• add-clean-maiyamok
• add-convert_currency
• add-cooldown
• add-copyright
• add-coref
• add-correct-engine
• add-count_thai_chars
• add-create_wordlist
• add-dependency-parser
• add-deprecated-word_vector
• add-dict-input-norvigspell
• add-editorconfig
• add-el
• add-encoding
• add-family-names
• add-find_synonym
• add-geo_neo
• add-han_solo
• add-hf-hub
• add-iso_11940-transliteration
• add-khavee-docs-tests
• add-live-dead-syllable-classify
• add-lst20-ner-onnx
• add-ltw2v-word_vector
• add-more-g2p
• add-more-lint-rules
• add-more-tests
• add-ner-class
• add-nercut
• add-new-misspelling
• add-new-science-word
• add-new-words
• add-nlpo3
• add-orst-list-and-change-correct
• add-perceptron-tagger
• add-phoneme-util
• add-pipline
• add-pypy-tox
• add-pythainlp-chat
• add-pythainlp.morpheme
• add-remove_tone_ipa
• add-retrained-crfcut-model
• add-rhyme
• add-save-load-param_free
• add-small100
• add-spdx-filetype
• add-spell_words
• add-spelling
• add-sponsors
• add-spoonerism
• add-syllable_tokenize
• add-tcc-test
• add-test-valueerror
• add-text-generator
• add-textaugment
• add-th_tdtb
• add-thai-lunar-date
• add-thai-morse
• add-thai-number2arabic-digit
• add-thai_nner
• add-thai_pangram
• add-thai_synonym
• add-thai_to_idn
• add-thai_w2p_v0.2
• add-thaig2p_v2
• add-thainer-model
• add-thainer-v2
• add-tltk
• add-tone-detector
• add-translate
• add-translate-class
• add-tud
• add-type-marker
• add-ud_goeswith
• add-umlfit-test
• add-update-dict
• add-windows-macos-ci
• add-word-dict
• add-word2word
• add-word_approximation
• add-word_detokenize
• add-wsd
• add-wtpsplit
• add-wunsen
• agents-md
• bact-patch-1
• bact-rm-example-com-test
• benchmark-to-cli
• better-readme
• bleuscore-type
• bump-530
• bump-532
• change-corpus-license
• change-coverage2github
• copilot/add-attaparse-dependency-parser
• copilot/add-bleu-rouge-metrics
• copilot/add-complete-soundex-thai
• copilot/add-lk82-soundex
• copilot/add-ner-thai-nner-dataset
• copilot/add-phupha-dataset
• copilot/add-profanity-detection-thai
• copilot/add-pythainlp-braille
• copilot/add-qwen3-0-6b-model
• copilot/add-static-type-check-workflow
• copilot/add-test-suite-for-pythainlp
• copilot/add-tests-for-output-coverage
• copilot/add-thai-consonant-clusters
• copilot/add-type-annotations-coverage
• copilot/add-type-hints-to-submodules-another-one
• copilot/add-type-hints-to-submodules-one-more-time
• copilot/add-type-hints-to-submodules-yet-again
• copilot/add-type-hints-to-tokenize
• copilot/add-typed-dicts-for-complex-dictionaries
• copilot/automate-docs-build-process
• copilot/bug-fix-pythainlp-path-manipulation
• copilot/check-distributed-operations-support
• copilot/check-docs-comments-code
• copilot/check-download-read-only-mode
• copilot/check-issue-1030-status
• copilot/check-thread-safeness-word-tokenize
• copilot/check-zip-safe-status
• copilot/cleanup-unused-code-and-docs
• copilot/exclude-words-in-translation
• copilot/fix-camembert-roberta-issue
• copilot/fix-check-marttra-bug
• copilot/fix-coverage-parser-error
• copilot/fix-db-json-creation-issue
• copilot/fix-duplicate-word-warning
• copilot/fix-inconsistent-type-hints
• copilot/fix-issue-887
• copilot/fix-job-cancellation-status
• copilot/fix-memory-usage-newmm-tokenization
• copilot/fix-mypy-issues
• copilot/fix-romanization-issues
• copilot/fix-romanize-empty-string
• copilot/fix-security-path-manipulation
• copilot/fix-str-default-parameter-usage
• copilot/fix-thai-word-tone-detector
• copilot/fix-thai-word-tone-detector-again
• copilot/fix-thai2rom-onnx-issue
• copilot/fix-transliterator-key-error
• copilot/fix-type-hints-inconsistencies
• copilot/fix-type-hints-inconsistent
• copilot/generate-citation-from-codemeta
• copilot/identify-caching-opportunities
• copilot/identify-security-issues
• copilot/improve-corpus-test-speed
• copilot/improve-english-in-docs
• copilot/improve-pythainlp-docs
• copilot/improve-pythainlp-performance
• copilot/increase-test-coverage
• copilot/incremental-type-annotations
• copilot/investigate-tokenization-issue
• copilot/narrow-type-hints-update
• copilot/optimize-unittest-workflow
• copilot/organise-test-cases-by-functions
• copilot/prepare-5-3-0-release
• copilot/reduce-dependencies-in-pythainlp
• copilot/refactor-any-type-annotations
• copilot/refactor-delayed-imports
• copilot/refactor-featurize-function
• copilot/refactor-inconsistent-naming-conventions
• copilot/refactor-noauto-test-suites
• copilot/remove-auto-download-corpus
• copilot/remove-star-imports
• copilot/replace-map-reduce-with-comprehension
• copilot/revert-pep604-union-types
• copilot/review-pr-changes
• copilot/scan-repository-type-hints
• copilot/scan-tests-extra-suite
• copilot/simplify-configuration-files
• copilot/tidy-code-for-consistency
• copilot/update-python-type-hints
• copilot/update-type-hint-analyser
• copilot/update-type-hint-analyzer
• copilot/update-version-and-docs
• copilot/use-pyicu-wheels-for-tests
• copilot/validate-example-code-docstrings
• copilot/verify-type-annotations
• copilot/verify-type-hints-and-docs
• copilot/verify-type-hints-annotations
• corpus-version
• delete-lst20-model
• dependabot/github_actions/actions/checkout-5
• dependabot/github_actions/actions/checkout-6
• dependabot/github_actions/actions/download-artifact-4
• dependabot/github_actions/actions/download-artifact-5
• dependabot/github_actions/actions/download-artifact-6
• dependabot/github_actions/actions/download-artifact-7
• dependabot/github_actions/actions/first-interaction-2
• dependabot/github_actions/actions/first-interaction-3
• dependabot/github_actions/actions/upload-artifact-4
• dependabot/github_actions/actions/upload-artifact-5
• dependabot/github_actions/actions/upload-artifact-6
• dependabot/github_actions/astral-sh/ruff-action-2
• dependabot/github_actions/astral-sh/ruff-action-3
• dependabot/github_actions/github/codeql-action-4
• dependabot/pip/bpemb-0.3.6
• dependabot/pip/emoji-0.6.0
• dependabot/pip/emoji-1.7.0
• dependabot/pip/emoji-2.14.0
• dependabot/pip/epitran-1.25.1
• dependabot/pip/epitran-1.26.0
• dependabot/pip/epitran-1.34.0
• dependabot/pip/esupar-1.6.9
• dependabot/pip/esupar-1.7.5
• dependabot/pip/fairseq-0.12.2
• dependabot/pip/fastai-2.6.3
• dependabot/pip/fastai-2.7.18
• dependabot/pip/gensim-4.3.3
• dependabot/pip/h5py-3.12.1
• dependabot/pip/h5py-3.13.0
• dependabot/pip/khanaa-0.1.1
• dependabot/pip/nltk-3.6.5
• dependabot/pip/numpy-eq-1.26.star
• dependabot/pip/numpy-eq-2.0.star
• dependabot/pip/numpy-eq-2.1.star
• dependabot/pip/numpy-eq-2.2.star
• dependabot/pip/numpy-gte-1.26.0-and-lt-3
• dependabot/pip/pandas-eq-2.2.star
• dependabot/pip/panphon-0.21.2
• dependabot/pip/panphon-0.22.2
• dependabot/pip/protobuf-3.20.2
• dependabot/pip/protobuf-5.29.1
• dependabot/pip/protobuf-5.29.2
• dependabot/pip/protobuf-5.29.3
• dependabot/pip/protobuf-6.30.1
• dependabot/pip/pyicu-2.14
• dependabot/pip/pyicu-2.15
• dependabot/pip/pyicu-2.15.1
• dependabot/pip/pyicu-2.15.2
• dependabot/pip/python-crfsuite-0.9.12
• dependabot/pip/pyyaml-6.0.2
• dependabot/pip/requests-eq-2.32.star
• dependabot/pip/sacremoses-0.1.1
• dependabot/pip/sentence-transformers-2.7.0
• dependabot/pip/sentence-transformers-3.2.1
• dependabot/pip/sentence-transformers-3.3.1
• dependabot/pip/sentence-transformers-4.0.1
• dependabot/pip/sentence-transformers-4.0.2
• dependabot/pip/sentence-transformers-4.1.0
• dependabot/pip/sentence-transformers-gte-2.7.0-and-lt-6
• dependabot/pip/sentencepiece-0.2.0
• dependabot/pip/spacy-eq-3.8.star
• dependabot/pip/spacy-thai-0.7.8
• dependabot/pip/spylls-0.1.7
• dependabot/pip/symspellpy-6.7.8
• dependabot/pip/symspellpy-6.9.0
• dependabot/pip/tensorflow-2.11.1
• dependabot/pip/tensorflow-2.18.0
• dependabot/pip/tensorflow-2.18.1
• dependabot/pip/tensorflow-2.4.2
• dependabot/pip/tensorflow-2.5.1
• dependabot/pip/tensorflow-2.5.2
• dependabot/pip/tensorflow-2.5.3
• dependabot/pip/tensorflow-2.6.4
• dependabot/pip/tensorflow-2.9.3
• dependabot/pip/torch-1.13.1
• dependabot/pip/torch-2.4.1
• dependabot/pip/torch-2.5.1
• dependabot/pip/transformers-4.30.0
• dependabot/pip/transformers-4.36.0
• dependabot/pip/transformers-4.38.0
• dependabot/pip/transformers-4.47.0
• dependabot/pip/transformers-4.47.1
• dependabot/pip/transformers-4.48.0
• dependabot/pip/transformers-4.48.1
• dependabot/pip/transformers-4.48.2
• dependabot/pip/transformers-4.48.3
• dependabot/pip/transformers-4.49.0
• dependabot/pip/transformers-4.50.3
• dependabot/pip/transformers-4.51.0
• dependabot/pip/transformers-4.51.3
• dependabot/pip/transformers-4.52.3
• dependabot/pip/transformers-4.53.0
• dependabot/pip/transformers-4.53.1
• dependabot/pip/transformers-4.53.2
• dependabot/pip/transformers-4.54.1
• dependabot/pip/transformers-4.55.0
• dependabot/pip/transformers-4.55.2
• dependabot/pip/transformers-4.55.4
• dependabot/pip/transformers-4.57.0
• dependabot/pip/transformers-4.57.1
• dependabot/pip/transformers-4.57.3
• dependabot/pip/transformers-4.57.5
• dependabot/pip/transformers-4.57.6
• dependabot/pip/ufal-chu-liu-edmonds-1.0.3
• dependabot/pip/wtpsplit-1.3.0
• dependabot/pip/wtpsplit-2.1.1
• deprecate-conceptnet
• dev
• display-thai-char
• docs-dep
• fix-1004
• fix-461
• fix-base-dependencies
• fix-ci-bug
• fix-corpus-utf8
• fix-deprecate-word-vec
• fix-dockerfile
• fix-etcc-dict
• fix-gensim
• fix-iso-11940-dup-keys
• fix-karan-tcc
• fix-keybert-type
• fix-lunar-last-day
• fix-maiyamok
• fix-markdown-issues
• fix-mypy-test
• fix-ruff-errors
• fix-symspellpy
• fix-test
• fix-thai_strftime
• fix-tokenization-benchmark
• fix-type-tests
• fix-types-test
• fix-umlfit
• fix-update-bug
• fix/help
• fix_docs_ci
• fixed-#609
• fixed-1055
• hotfix-bug-normalize
• improve-1
• improve-w2p
• korakot-patch-1
• limit-unittest-trigger
• limiting-unitttest
• lint-tests
• llm-tool
• merge-syllable-subword
• minor-tidy
• more-tests
• move-model
• nerentity
• patch-1110
• prepare-5-3-3-release
• protect-literal-eval
• pythainlp-3.0
• pyup-config
• reduce-peak-memory
• remotes/origin/add-family-names
• remotes/origin/korakot-patch-1
• remove-act-bin
• remove-clause_tokenize
• remove-code-encodings
• remove-conceptnet
• remove-deprecated-function
• remove-pickle
• remove-spylls
• retrain-ud-pud-postag
• rm-mypy-overrides
• rm-trailing-spaces-volubilis
• rm-trailing-whitespaces
• simpify-calculate_f_year_f_dev
• start-2.4
• start-3.0
• start-beta1
• test-cff-generation-workflow
• text-cleaning
• thai-ner-1-5-2
• thai-readme
• tox-pyproject
• unittest-main
• update-actions-version
• update-attacut
• update-docker-fairseq
• update-docs
• update-import
• update-import-2.2
• update-import-python-crfsuite
• update-maiyamok
• update-nlpo3-python-v1.2.2
• update-postag-docs
• update-pythainlp-download-corpus
• update-royin
• use-hatchling
• v2.3.2
• v2.4.0-dev0
• v3.0.0
• v3.0.0-beta
• v3.0.0-dev0
• v3.0.1
• v3.0.10
• v3.0.2
• v3.0.3
• v3.0.5
• v3.0.6
• v3.0.9
• v3.1.0
• v3.1.0-beta0
• v3.1.0-dev0
• v3.1.0-dev1
• v3.1.0-dev2
• v3.1.0-dev3
• v3.1.1
• v4.0.0
• v4.0.0-beta1
• v4.0.2
• v4.1.0-beta1
• v4.1.0-beta2
• v4.1.0-beta3
• v4.1.0-beta5
• v5.0.0
• v5.0.0-beta1
• v5.0.0-dev0
• v5.0.0-dev1
• v5.0.0-dev2
• v5.0.1
• v5.0.2
• v5.0.3
• v5.0.4
• v5.1.0
• v5.1.0-beta1
• v5.1.1
• v5.1.2
• v5.2.0
• v5.2.0-beta1
• wannaphong/add-codemeta
• wannaphong/add-display-cell-tokenizer
• wannaphong/add-lcs
• wannaphong/add-misspell-command
• wannaphong/fix-custom-dict-error
• wannaphong/fix-newmm-safe
• wannaphong/fix-tokenizer

Build # 21246405491

Build Type

Pull #1222

github

Committed by Copilot

Commit Message

Make normalization more conservative to avoid creating new words

- Update remove_spaces_before_marks() to only remove space after consonants not preceded by vowels
- Add removal of dangling chars after spaces in remove_dangling()
- Remove unused import remove_spaces_before_marks from tests
- Fix import ordering per ruff recommendations

Co-authored-by: bact <128572+bact@users.noreply.github.com>

Pull Request Pull Request #1222: Fix normalize() to remove spaces before Thai tone marks and non-base characters (conservative approach)

Run Details

9 of 9 new or added lines in 1 file covered. (100.0%)

3 existing lines in 1 file now uncovered.

4557 of 8493 relevant lines covered (53.66%)

0.54 hits per line