PyThaiNLP / pythainlp / 23082486900

66%

Build Type

push

github

Committed by web-flow

Commit Message

Use npz instead of pickle for Thai W2P weights (#1328)

* Update np.load to disallow allow_pickle

Change np.load to disallow pickling for security.

* Fix np.load allow_pickle=False to work correctly with .npz format (#1329)

* Initial plan

* Fix np.load allow_pickle=False to work with .npz NpzFile format

- Replace .item().get(key) with [key] dict-style access on NpzFile
- Remove variables instance attribute; use local variable instead
- Add type annotation for variables local var as np.lib.npyio.NpzFile
- Add allow_pickle=False to embeddings.npy load in words_spelling_correction.py

Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com>

* Initial plan

* Fix np.load allow_pickle for legacy .npy corpus (ValueError fix)

The thai_w2p corpus v0.2 is stored as a .npy pickled dict.
Loading it with allow_pickle=False raises ValueError.

Detect file format by extension:
- .npz: use allow_pickle=False (secure, for future corpus versions)
- .npy (legacy): use allow_pickle=True + dict validation

Also add `import os` for os.path.splitext().

Co-authored-by: wannaphong <8536487+wannaphong@users.noreply.github.com>

* Initial plan

* Sync dev, add pickle warning, fix docstring and code style

Co-authored-by: bact <128572+bact@users.noreply.github.com>

* Update CHANGELOG for release 5.3.1

This release focuses on security issues related to corpus file loading, including improved pickle handling and defensive file loading.

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <1... (continued)

Run Details

22 of 23 new or added lines in 5 files covered. (95.65%)

2 existing lines in 2 files now uncovered.

6386 of 9686 relevant lines covered (65.93%)

0.66 hits per line