3562198437

Build # 3562198437

Build Type

github

Committed by GitHub

Commit Message

<a href="https://github.com/PyThaiNLP/pythainlp/commit/<a class=hub.com/PyThaiNLP/pythainlp/commit/32018b6e74dd25d47f59dc0ef4ed01eba77bf655">32018b6e7<a href="https://github.com/PyThaiNLP/pythainlp/commit/32018b6e74dd25d47f59dc0ef4ed01eba77bf655">">Merge </a><a class="double-link" href="https://github.com/PyThaiNLP/pythainlp/commit/<a class="double-link" href="https://github.com/PyThaiNLP/pythainlp/commit/659753527f467d7c65f7a3e728341174fd8ba0f0">659753527</a>">659753527</a><a href="https://github.com/PyThaiNLP/pythainlp/commit/32018b6e74dd25d47f59dc0ef4ed01eba77bf655"> into cfde2edd2">cfde2edd2</a>

Pull Request Pull Request #757: [WIP] Add ud_goeswith

Run Details

52 of 52 new or added lines in 2 files covered. (100.0%)

5050 of 5708 relevant lines covered (88.47%)

0.88 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

35.0

/pythainlp/parse/core.py

# -*- coding: utf-8 -*-
from typing import List, Union


_tagger = None
_tagger_name = ""


def dependency_parsing(
    text: str, model: str = None, tag: str = "str", engine: str = "esupar"
) -> Union[List[List[str]], str]:
    """
    Dependency Parsing

    :param str text: text to do dependency parsing
    :param str model: model for using with engine \
        (for esupar and transformers_ud)
    :param str tag: output type (str or list)
    :param str engine: the name dependency parser
    :return: str (conllu) or List
    :rtype: Union[List[List[str]], str]

    **Options for engine**
        * *esupar* (default) - Tokenizer POS-tagger and Dependency-parser \
            with BERT/RoBERTa/DeBERTa model. `GitHub \
                <https://github.com/KoichiYasuoka/esupar>`_
        * *spacy_thai* - Tokenizer, POS-tagger, and dependency-parser \
            for Thai language, working on Universal Dependencies. \
            `GitHub <https://github.com/KoichiYasuoka/spacy-thai>`_
        * *transformers_ud* - TransformersUD \
            `GitHub <https://github.com/KoichiYasuoka/>`_
        * *ud_goeswith* - POS-tagging and dependency-parsing with \
            using `goeswith` for subwords

    **Options for model (esupar engine)**
        * *th* (default) - KoichiYasuoka/roberta-base-thai-spm-upos model \
            `Huggingface \
            <https://huggingface.co/KoichiYasuoka/roberta-base-thai-spm-upos>`_
        * *KoichiYasuoka/deberta-base-thai-upos* - DeBERTa(V2) model \
            pre-trained on Thai Wikipedia texts for POS-tagging and \
            dependency-parsing `Huggingface \
            <https://huggingface.co/KoichiYasuoka/deberta-base-thai-upos>`_
        * *KoichiYasuoka/roberta-base-thai-syllable-upos* - RoBERTa model \
            pre-trained on Thai Wikipedia texts for POS-tagging and \
            dependency-parsing. (syllable level) `Huggingface \
            <https://huggingface.co/KoichiYasuoka/roberta-base-thai-syllable-upos>`_
        * *KoichiYasuoka/roberta-base-thai-char-upos* - RoBERTa model \
            pre-trained on Thai Wikipedia texts for POS-tagging \
            and dependency-parsing. (char level) `Huggingface \
            <https://huggingface.co/KoichiYasuoka/roberta-base-thai-char-upos>`_

    If you want to train model for esupar, you can read \
    `Huggingface <https://github.com/KoichiYasuoka/esupar>`_

    **Options for model (transformers_ud engine)**
        * *KoichiYasuoka/deberta-base-thai-ud-head* (default) - \
            DeBERTa(V2) model pretrained on Thai Wikipedia texts \
            for dependency-parsing (head-detection on Universal \
            Dependencies) as question-answering, derived from \
            deberta-base-thai. \
            trained by th_blackboard.conll. `Huggingface \
            <https://huggingface.co/KoichiYasuoka/deberta-base-thai-ud-head>`_
        * *KoichiYasuoka/roberta-base-thai-spm-ud-head* - \
            roberta model pretrained on Thai Wikipedia texts \
            for dependency-parsing. `Huggingface \
            <https://huggingface.co/KoichiYasuoka/roberta-base-thai-spm-ud-head>`_

    **Options for model (ud_goeswith engine)**
        * *KoichiYasuoka/deberta-base-thai-ud-goeswith* (default) - \
            This is a DeBERTa(V2) model pre-trained on Thai Wikipedia \
            texts for POS-tagging and dependency-parsing (using goeswith for subwords) \
            `Huggingface <https://huggingface.co/KoichiYasuoka/deberta-base-thai-ud-goeswith>`_

    :Example:
    ::

        from pythainlp.parse import dependency_parsing

        print(dependency_parsing("ผมเป็นคนดี", engine="esupar"))
        # output:
        # 1       ผม      _       PRON    _       _       3       nsubj   _       SpaceAfter=No
        # 2       เป็น     _       VERB    _       _       3       cop     _       SpaceAfter=No
        # 3       คน      _       NOUN    _       _       0       root    _       SpaceAfter=No
        # 4       ดี       _       VERB    _       _       3       acl     _       SpaceAfter=No

        print(dependency_parsing("ผมเป็นคนดี", engine="spacy_thai"))
        # output:
        # 1       ผม              PRON    PPRS    _       2       nsubj   _       SpaceAfter=No
        # 2       เป็น             VERB    VSTA    _       0       ROOT    _       SpaceAfter=No
        # 3       คนดี             NOUN    NCMN    _       2       obj     _       SpaceAfter=No
    """
    global _tagger, _tagger_name
    if _tagger_name != engine:
        if engine == "esupar":
            from pythainlp.parse.esupar_engine import Parse

            _tagger = Parse(model=model)
        elif engine == "transformers_ud":
            from pythainlp.parse.transformers_ud import Parse

            _tagger = Parse(model=model)
        elif engine == "spacy_thai":
            from pythainlp.parse.spacy_thai_engine import Parse

            _tagger = Parse()
        elif engine == "ud_goeswith":
            from pythainlp.parse.ud_goeswith import Parse

            _tagger = Parse(model=model)
        else:
            raise NotImplementedError("The engine doesn't support.")
    _tagger_name = engine
    return _tagger(text, tag=tag)

1	# -- coding: utf-8 --
2	from typing import List, Union	1✔
3
4
5	_tagger = None	1✔
6	_tagger_name = ""	1✔
7
8
9	def dependency_parsing(	1✔
10	text: str, model: str = None, tag: str = "str", engine: str = "esupar"
11	) -> Union[List[List[str]], str]:
12	"""
13	Dependency Parsing
14
15	:param str text: text to do dependency parsing
16	:param str model: model for using with engine \
17	(for esupar and transformers_ud)
18	:param str tag: output type (str or list)
19	:param str engine: the name dependency parser
20	:return: str (conllu) or List
21	:rtype: Union[List[List[str]], str]
22
23	Options for engine
24	* esupar (default) - Tokenizer POS-tagger and Dependency-parser \
25	with BERT/RoBERTa/DeBERTa model. `GitHub \
26	<https://github.com/KoichiYasuoka/esupar>`_
27	* spacy_thai - Tokenizer, POS-tagger, and dependency-parser \
28	for Thai language, working on Universal Dependencies. \
29	`GitHub <https://github.com/KoichiYasuoka/spacy-thai>`_
30	* transformers_ud - TransformersUD \
31	`GitHub <https://github.com/KoichiYasuoka/>`_
32	* ud_goeswith - POS-tagging and dependency-parsing with \
33	using `goeswith` for subwords
34
35	Options for model (esupar engine)
36	* th (default) - KoichiYasuoka/roberta-base-thai-spm-upos model \
37	`Huggingface \
38	<https://huggingface.co/KoichiYasuoka/roberta-base-thai-spm-upos>`_
39	* KoichiYasuoka/deberta-base-thai-upos - DeBERTa(V2) model \
40	pre-trained on Thai Wikipedia texts for POS-tagging and \
41	dependency-parsing `Huggingface \
42	<https://huggingface.co/KoichiYasuoka/deberta-base-thai-upos>`_
43	* KoichiYasuoka/roberta-base-thai-syllable-upos - RoBERTa model \
44	pre-trained on Thai Wikipedia texts for POS-tagging and \
45	dependency-parsing. (syllable level) `Huggingface \
46	<https://huggingface.co/KoichiYasuoka/roberta-base-thai-syllable-upos>`_
47	* KoichiYasuoka/roberta-base-thai-char-upos - RoBERTa model \
48	pre-trained on Thai Wikipedia texts for POS-tagging \
49	and dependency-parsing. (char level) `Huggingface \
50	<https://huggingface.co/KoichiYasuoka/roberta-base-thai-char-upos>`_
51
52	If you want to train model for esupar, you can read \
53	`Huggingface <https://github.com/KoichiYasuoka/esupar>`_
54
55	Options for model (transformers_ud engine)
56	* KoichiYasuoka/deberta-base-thai-ud-head (default) - \
57	DeBERTa(V2) model pretrained on Thai Wikipedia texts \
58	for dependency-parsing (head-detection on Universal \
59	Dependencies) as question-answering, derived from \
60	deberta-base-thai. \
61	trained by th_blackboard.conll. `Huggingface \
62	<https://huggingface.co/KoichiYasuoka/deberta-base-thai-ud-head>`_
63	* KoichiYasuoka/roberta-base-thai-spm-ud-head - \
64	roberta model pretrained on Thai Wikipedia texts \
65	for dependency-parsing. `Huggingface \
66	<https://huggingface.co/KoichiYasuoka/roberta-base-thai-spm-ud-head>`_
67
68	Options for model (ud_goeswith engine)
69	* KoichiYasuoka/deberta-base-thai-ud-goeswith (default) - \
70	This is a DeBERTa(V2) model pre-trained on Thai Wikipedia \
71	texts for POS-tagging and dependency-parsing (using goeswith for subwords) \
72	`Huggingface <https://huggingface.co/KoichiYasuoka/deberta-base-thai-ud-goeswith>`_
73
74	:Example:
75	::
76
77	from pythainlp.parse import dependency_parsing
78
79	print(dependency_parsing("ผมเป็นคนดี", engine="esupar"))
80	# output:
81	# 1 ผม _ PRON _ _ 3 nsubj _ SpaceAfter=No
82	# 2 เป็น _ VERB _ _ 3 cop _ SpaceAfter=No
83	# 3 คน _ NOUN _ _ 0 root _ SpaceAfter=No
84	# 4 ดี _ VERB _ _ 3 acl _ SpaceAfter=No
85
86	print(dependency_parsing("ผมเป็นคนดี", engine="spacy_thai"))
87	# output:
88	# 1 ผม PRON PPRS _ 2 nsubj _ SpaceAfter=No
89	# 2 เป็น VERB VSTA _ 0 ROOT _ SpaceAfter=No
90	# 3 คนดี NOUN NCMN _ 2 obj _ SpaceAfter=No
91	"""
92	global _tagger, _tagger_name
93	if _tagger_name != engine:	1✔
94	if engine == "esupar":	1✔
95	from pythainlp.parse.esupar_engine import Parse	1✔
96
97	_tagger = Parse(model=model)	×
98	elif engine == "transformers_ud":	×
99	from pythainlp.parse.transformers_ud import Parse	×
100
101	_tagger = Parse(model=model)	×
102	elif engine == "spacy_thai":	×
103	from pythainlp.parse.spacy_thai_engine import Parse	×
104
105	_tagger = Parse()	×
106	elif engine == "ud_goeswith":	×
107	from pythainlp.parse.ud_goeswith import Parse	×
108
109	_tagger = Parse(model=model)	×
110	else:
111	raise NotImplementedError("The engine doesn't support.")	×
112	_tagger_name = engine	×
113	return _tagger(text, tag=tag)	×

PyThaiNLP / pythainlp / 3562198437

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous