2114953157

Build Type

github

Committed by GitHub

Commit Message

<a href="https://github.com/JohnSnowLabs/spark-nlp/commit/<a class=hub.com/JohnSnowLabs/spark-nlp/commit/<a class="double-link" href="https://git"><a class=hub.com/JohnSnowLabs/spark-nlp/commit/<a class="double-link" href="https://git"><a class=hub.com/JohnSnowLabs/spark-nlp/commit/12d5b8bb41811c48de270945704655193b1bd38a">12d5b8bb4">&lt;a href=&quot;https://github.com/JohnSnowLabs/spark-nlp/commit/</a><a class="double-link" href="https://github.com/JohnSnowLabs/spark-nlp/commit/&lt;a class=&quot;double-link&quot; href=&quot;https://git">&lt;a class=</a>hub.com/JohnSnowLabs/spark-nlp/commit/&lt;a class=&quot;double-link&quot; href=&quot;https://git">&lt;a class=</a>hub.com/JohnSnowLabs/spark-nlp/commit/&lt;a class=&quot;double-link&quot; href=&quot;https://git">&lt;a class=</a>hub.com/JohnSnowLabs/spark-nlp/commit/&lt;a class=&quot;double-link&quot; href=&quot;https://git">&lt;a class=</a>hub.com/JohnSnowLabs/spark-nlp/commit/&lt;a class=&quot;double-link&quot; href=&quot;https://git">&lt;a class=</a>hub.com/JohnSnowLabs/spark-nlp/commit/12d5b8bb41811c48de270945704655193b1bd38a">12d5b8bb4</a><a href="https://github.com/JohnSnowLabs/spark-nlp/commit/12d5b8bb41811c48de270945704655193b1bd38a">&quot;&gt;&amp;lt;a href=&amp;quot;https://github.com/JohnSnowLabs/spark-nlp/commit/&lt;/a&gt;&lt;a class=&quot;double-link&quot; href=&quot;https://github.com/JohnSnowLabs/spark-nlp/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/JohnSnowLabs/spark-nlp/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/JohnSnowLabs/spark-nlp/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/JohnSnowLabs/spark-nlp/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/JohnSnowLabs/spark-nlp/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/JohnSnowLabs/spark-nlp/commit/12d5b8bb41811c48de270945704655193b1bd38a&quot;&gt;12d5b8bb4&lt;/a&gt;&lt;a href=&q... (continued)

Pull Request Pull Request #7234: Release/343 release candidate

Run Details

4001 of 4001 new or added lines in 186 files covered. (100.0%)

8081 of 10941 relevant lines covered (73.86%)

0.74 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

61.54

/src/main/scala/com/johnsnowlabs/nlp/annotators/ld/dl/LanguageDetectorDL.scala

/*
 * Copyright 2017-2022 John Snow Labs
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package com.johnsnowlabs.nlp.annotators.ld.dl

import com.johnsnowlabs.ml.tensorflow._
import com.johnsnowlabs.nlp._
import com.johnsnowlabs.nlp.annotators.common._
import com.johnsnowlabs.nlp.serialization.MapFeature
import com.johnsnowlabs.nlp.util.io.{ExternalResource, ReadAs, ResourceHelper}
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.ml.param._
import org.apache.spark.ml.util.Identifiable
import org.apache.spark.sql.SparkSession

import java.io.File
import scala.collection.immutable.ListMap

/** Language Identification and Detection by using CNN and RNN architectures in TensorFlow.
  *
  * `LanguageDetectorDL` is an annotator that detects the language of documents or sentences
  * depending on the inputCols. The models are trained on large datasets such as Wikipedia and
  * Tatoeba. Depending on the language (how similar the characters are), the LanguageDetectorDL
  * works best with text longer than 140 characters. The output is a language code in
  * [[https://en.wikipedia.org/wiki/List_of_Wikipedias Wiki Code style]].
  *
  * Pretrained models can be loaded with `pretrained` of the companion object:
  * {{{
  * Val languageDetector = LanguageDetectorDL.pretrained()
  *   .setInputCols("sentence")
  *   .setOutputCol("language")
  * }}}
  * The default model is `"ld_wiki_tatoeba_cnn_21"`, default language is `"xx"` (meaning
  * multi-lingual), if no values are provided. For available pretrained models please see the
  * [[https://nlp.johnsnowlabs.com/models?task=Language+Detection Models Hub]].
  *
  * For extended examples of usage, see the
  * [[https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/annotation/english/language-detection/Language_Detection_and_Indentification.ipynb Spark NLP Workshop]]
  * And the
  * [[https://github.com/JohnSnowLabs/spark-nlp/blob/master/src/test/scala/com/johnsnowlabs/nlp/annotators/ld/dl/LanguageDetectorDLTestSpec.scala LanguageDetectorDLTestSpec]].
  *
  * ==Example==
  * {{{
  * import spark.implicits._
  * import com.johnsnowlabs.nlp.base.DocumentAssembler
  * import com.johnsnowlabs.nlp.annotators.ld.dl.LanguageDetectorDL
  * import org.apache.spark.ml.Pipeline
  *
  * val documentAssembler = new DocumentAssembler()
  *   .setInputCol("text")
  *   .setOutputCol("document")
  *
  * val languageDetector = LanguageDetectorDL.pretrained()
  *   .setInputCols("document")
  *   .setOutputCol("language")
  *
  * val pipeline = new Pipeline()
  *   .setStages(Array(
  *     documentAssembler,
  *     languageDetector
  *   ))
  *
  * val data = Seq(
  *   "Spark NLP is an open-source text processing library for advanced natural language processing for the Python, Java and Scala programming languages.",
  *   "Spark NLP est une bibliothèque de traitement de texte open source pour le traitement avancé du langage naturel pour les langages de programmation Python, Java et Scala.",
  *   "Spark NLP ist eine Open-Source-Textverarbeitungsbibliothek für fortgeschrittene natürliche Sprachverarbeitung für die Programmiersprachen Python, Java und Scala."
  * ).toDF("text")
  * val result = pipeline.fit(data).transform(data)
  *
  * result.select("language.result").show(false)
  * +------+
  * |result|
  * +------+
  * |[en]  |
  * |[fr]  |
  * |[de]  |
  * +------+
  * }}}
  *
  * @groupname anno Annotator types
  * @groupdesc anno
  *   Required input and expected output annotator types
  * @groupname Ungrouped Members
  * @groupname param Parameters
  * @groupname setParam Parameter setters
  * @groupname getParam Parameter getters
  * @groupname Ungrouped Members
  * @groupprio param  1
  * @groupprio anno  2
  * @groupprio Ungrouped 3
  * @groupprio setParam  4
  * @groupprio getParam  5
  * @groupdesc param
  *   A list of (hyper-)parameter keys this annotator can take. Users can set and get the
  *   parameter values through setters and getters, respectively.
  */
class LanguageDetectorDL(override val uid: String)
    extends AnnotatorModel[LanguageDetectorDL]
    with HasSimpleAnnotate[LanguageDetectorDL]
    with WriteTensorflowModel {

  def this() = this(Identifiable.randomUID("LANGUAGE_DETECTOR_DL"))

  /** Alphabet used to feed the TensorFlow model for prediction
    *
    * @group param
    */
  val alphabet: MapFeature[String, Int] = new MapFeature(this, "alphabet")

  /** Language used to map prediction to ISO 639-1 language codes
    *
    * @group param
    */
  val language: MapFeature[String, Int] = new MapFeature(this, "language")

  /** The minimum threshold for the final result, otherwise it will be either `"unk"` or the value
    * set in `thresholdLabel` (Default: `0.1f`). Value is between 0.0 to 1.0. Try to set this
    * lower if your text is hard to predict
    *
    * @group param
    */
  val threshold = new FloatParam(
    this,
    "threshold",
    "The minimum threshold for the final result otherwise it will be either Unknown or the value set in thresholdLabel.")

  /** Value for the classification, if confidence is less than `threshold` (Default: `"unk"`).
    *
    * @group param
    */
  val thresholdLabel = new Param[String](
    this,
    "thresholdLabel",
    "In case the score is less than threshold, what should be the label. Default is Unknown.")

  /** Output average of sentences instead of one output per sentence (Default: `true`).
    *
    * @group param
    */
  val coalesceSentences = new BooleanParam(
    this,
    "coalesceSentences",
    "If sets to true the output of all sentences will be averaged to one output instead of one output per sentence. Default to true.")

  /** ConfigProto from tensorflow, serialized into byte array. Get with
    * config_proto.SerializeToString()
    *
    * @group param
    */
  val configProtoBytes = new IntArrayParam(
    this,
    "configProtoBytes",
    "ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()")

  /** Languages the model was trained with.
    *
    * @group param
    */
  val languages =
    new StringArrayParam(this, "languages", "keep an internal copy of languages for Python")

  /** @group setParam */
  def setLanguage(value: Map[String, Int]): this.type = {
    if (get(language).isEmpty)
      set(this.language, value)
    this
  }

  /** @group setParam */
  def setAlphabet(value: Map[String, Int]): this.type = {
    if (get(language).isEmpty)
      set(alphabet, value)
    this
  }

  /** @group setParam */
  def setThreshold(threshold: Float): this.type = set(this.threshold, threshold)

  /** @group setParam */
  def setThresholdLabel(label: String): this.type = set(this.thresholdLabel, label)

  /** @group setParam */
  def setCoalesceSentences(value: Boolean): this.type = set(coalesceSentences, value)

  /** @group setParam */
  def setConfigProtoBytes(bytes: Array[Int]): LanguageDetectorDL.this.type =
    set(this.configProtoBytes, bytes)

  /** @group getParam */
  def getLanguage: Array[String] = {
    val langs = $$(language).keys.toArray
    set(languages, langs)
    langs
  }

  /** @group getParam */
  def getThreshold: Float = $(this.threshold)

  /** @group getParam */
  def getThresholdLabel: String = $(this.thresholdLabel)

  /** @group getParam */
  def getCoalesceSentences: Boolean = $(coalesceSentences)

  /** @group getParam */
  def getConfigProtoBytes: Option[Array[Byte]] = get(this.configProtoBytes).map(_.map(_.toByte))

  setDefault(
    inputCols -> Array("document"),
    outputCol -> "language",
    threshold -> 0.1f,
    thresholdLabel -> "unk",
    coalesceSentences -> true)

  private var _model: Option[Broadcast[TensorflowLD]] = None

  /** @group getParam */
  def getModelIfNotSet: TensorflowLD = _model.get.value

  /** @group setParam */
  def setModelIfNotSet(spark: SparkSession, tensorflow: TensorflowWrapper): this.type = {
    if (_model.isEmpty) {

      _model = Some(
        spark.sparkContext.broadcast(
          new TensorflowLD(
            tensorflow,
            configProtoBytes = getConfigProtoBytes,
            ListMap($$(language).toSeq.sortBy(_._2): _*),
            ListMap($$(alphabet).toSeq.sortBy(_._2): _*))))
    }

    this
  }

  /** Takes a document and annotations and produces new annotations of this annotator's annotation
    * type
    *
    * @param annotations
    *   Annotations that correspond to inputAnnotationCols generated by previous annotators if any
    * @return
    *   any number of annotations processed for every input annotation. Not necessary one to one
    *   relationship
    */
  override def annotate(annotations: Seq[Annotation]): Seq[Annotation] = {
    val sentences = SentenceSplit.unpack(annotations)
    val nonEmptySentences = sentences.filter(_.content.nonEmpty)
    if (nonEmptySentences.nonEmpty) {
      getModelIfNotSet.predict(
        nonEmptySentences,
        $(threshold),
        $(thresholdLabel),
        $(coalesceSentences))
    } else {
      Seq.empty[Annotation]
    }
  }

  /** Annotator reference id. Used to identify elements in metadata or to refer to this annotator
    * type
    */
  override val inputAnnotatorTypes: Array[String] = Array(AnnotatorType.DOCUMENT)
  override val outputAnnotatorType: AnnotatorType = AnnotatorType.LANGUAGE

  override def onWrite(path: String, spark: SparkSession): Unit = {
    super.onWrite(path, spark)
    writeTensorflowModelV2(
      path,
      spark,
      getModelIfNotSet.tensorflow,
      "_ld",
      LanguageDetectorDL.tfFile,
      configProtoBytes = getConfigProtoBytes)
  }

}

trait ReadablePretrainedLanguageDetectorDLModel
    extends ParamsAndFeaturesReadable[LanguageDetectorDL]
    with HasPretrained[LanguageDetectorDL] {
  override val defaultModelName: Some[String] = Some("ld_wiki_tatoeba_cnn_21")
  override val defaultLang: String = "xx"

  /** Java compliant-overrides */
  override def pretrained(): LanguageDetectorDL = super.pretrained()

  override def pretrained(name: String): LanguageDetectorDL = super.pretrained(name)

  override def pretrained(name: String, lang: String): LanguageDetectorDL =
    super.pretrained(name, lang)

  override def pretrained(name: String, lang: String, remoteLoc: String): LanguageDetectorDL =
    super.pretrained(name, lang, remoteLoc)
}

trait ReadLanguageDetectorDLTensorflowModel extends ReadTensorflowModel {
  this: ParamsAndFeaturesReadable[LanguageDetectorDL] =>

  override val tfFile: String = "ld_tensorflow"

  def readTensorflow(instance: LanguageDetectorDL, path: String, spark: SparkSession): Unit = {

    val tf = readTensorflowModel(path, spark, "_ld_tf")
    instance.setModelIfNotSet(spark, tf)
    // This allows for Python to access getLanguages function
    val t = instance.language.get.toArray
    val r = t(0).keys.toArray
    instance.set(instance.languages, r)
  }

  addReader(readTensorflow)

  def loadSavedModel(folder: String, spark: SparkSession): LanguageDetectorDL = {

    val f = new File(folder)
    val savedModel = new File(folder, "saved_model.pb")
    require(f.exists, s"Folder $folder not found")
    require(f.isDirectory, s"File $folder is not folder")
    require(savedModel.exists(), s"savedModel file saved_model.pb not found in folder $folder")

    val alphabetPath = new File(folder + "/assets", "alphabet.txt")
    val languagePath = new File(folder + "/assets", "language.txt")

    require(f.exists, s"Folder $folder not found")
    require(f.isDirectory, s"File $folder is not folder")
    require(alphabetPath.exists(), s"Alphabet file alphabet.txt not found in folder $folder")
    require(languagePath.exists(), s"Language file language.txt not found in folder $folder")

    val alphabetResource =
      new ExternalResource(alphabetPath.getAbsolutePath, ReadAs.TEXT, Map("format" -> "text"))
    val alphabets = ResourceHelper.parseLines(alphabetResource).zipWithIndex.toMap

    val languageResource =
      new ExternalResource(languagePath.getAbsolutePath, ReadAs.TEXT, Map("format" -> "text"))
    val languages = ResourceHelper.parseLines(languageResource).zipWithIndex.toMap

    val (wrapper, _) =
      TensorflowWrapper.read(folder, zipped = false, useBundle = true, tags = Array("serve"))

    new LanguageDetectorDL()
      .setAlphabet(alphabets)
      .setLanguage(languages)
      .setModelIfNotSet(spark, wrapper)
  }
}

/** This is the companion object of [[LanguageDetectorDL]]. Please refer to that class for the
  * documentation.
  */
object LanguageDetectorDL
    extends ReadablePretrainedLanguageDetectorDLModel
    with ReadLanguageDetectorDLTensorflowModel

1	/*
2	* Copyright 2017-2022 John Snow Labs
3	*
4	* Licensed under the Apache License, Version 2.0 (the "License");
5	* you may not use this file except in compliance with the License.
6	* You may obtain a copy of the License at
7	*
8	* http://www.apache.org/licenses/LICENSE-2.0
9	*
10	* Unless required by applicable law or agreed to in writing, software
11	* distributed under the License is distributed on an "AS IS" BASIS,
12	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13	* See the License for the specific language governing permissions and
14	* limitations under the License.
15	*/
16
17	package com.johnsnowlabs.nlp.annotators.ld.dl
18
19	import com.johnsnowlabs.ml.tensorflow._
20	import com.johnsnowlabs.nlp._
21	import com.johnsnowlabs.nlp.annotators.common._
22	import com.johnsnowlabs.nlp.serialization.MapFeature
23	import com.johnsnowlabs.nlp.util.io.{ExternalResource, ReadAs, ResourceHelper}
24	import org.apache.spark.broadcast.Broadcast
25	import org.apache.spark.ml.param._
26	import org.apache.spark.ml.util.Identifiable
27	import org.apache.spark.sql.SparkSession
28
29	import java.io.File
30	import scala.collection.immutable.ListMap
31
32	/** Language Identification and Detection by using CNN and RNN architectures in TensorFlow.
33	*
34	* `LanguageDetectorDL` is an annotator that detects the language of documents or sentences
35	* depending on the inputCols. The models are trained on large datasets such as Wikipedia and
36	* Tatoeba. Depending on the language (how similar the characters are), the LanguageDetectorDL
37	* works best with text longer than 140 characters. The output is a language code in
38	* [[https://en.wikipedia.org/wiki/List_of_Wikipedias Wiki Code style]].
39	*
40	* Pretrained models can be loaded with `pretrained` of the companion object:
41	* {{{
42	* Val languageDetector = LanguageDetectorDL.pretrained()
43	* .setInputCols("sentence")
44	* .setOutputCol("language")
45	* }}}
46	* The default model is `"ld_wiki_tatoeba_cnn_21"`, default language is `"xx"` (meaning
47	* multi-lingual), if no values are provided. For available pretrained models please see the
48	* [[https://nlp.johnsnowlabs.com/models?task=Language+Detection Models Hub]].
49	*
50	* For extended examples of usage, see the
51	* [[https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/annotation/english/language-detection/Language_Detection_and_Indentification.ipynb Spark NLP Workshop]]
52	* And the
53	* [[https://github.com/JohnSnowLabs/spark-nlp/blob/master/src/test/scala/com/johnsnowlabs/nlp/annotators/ld/dl/LanguageDetectorDLTestSpec.scala LanguageDetectorDLTestSpec]].
54	*
55	* ==Example==
56	* {{{
57	* import spark.implicits._
58	* import com.johnsnowlabs.nlp.base.DocumentAssembler
59	* import com.johnsnowlabs.nlp.annotators.ld.dl.LanguageDetectorDL
60	* import org.apache.spark.ml.Pipeline
61	*
62	* val documentAssembler = new DocumentAssembler()
63	* .setInputCol("text")
64	* .setOutputCol("document")
65	*
66	* val languageDetector = LanguageDetectorDL.pretrained()
67	* .setInputCols("document")
68	* .setOutputCol("language")
69	*
70	* val pipeline = new Pipeline()
71	* .setStages(Array(
72	* documentAssembler,
73	* languageDetector
74	* ))
75	*
76	* val data = Seq(
77	* "Spark NLP is an open-source text processing library for advanced natural language processing for the Python, Java and Scala programming languages.",
78	* "Spark NLP est une bibliothèque de traitement de texte open source pour le traitement avancé du langage naturel pour les langages de programmation Python, Java et Scala.",
79	* "Spark NLP ist eine Open-Source-Textverarbeitungsbibliothek für fortgeschrittene natürliche Sprachverarbeitung für die Programmiersprachen Python, Java und Scala."
80	* ).toDF("text")
81	* val result = pipeline.fit(data).transform(data)
82	*
83	* result.select("language.result").show(false)
84	* +------+
85	* \|result\|
86	* +------+
87	* \|[en] \|
88	* \|[fr] \|
89	* \|[de] \|
90	* +------+
91	* }}}
92	*
93	* @groupname anno Annotator types
94	* @groupdesc anno
95	* Required input and expected output annotator types
96	* @groupname Ungrouped Members
97	* @groupname param Parameters
98	* @groupname setParam Parameter setters
99	* @groupname getParam Parameter getters
100	* @groupname Ungrouped Members
101	* @groupprio param 1
102	* @groupprio anno 2
103	* @groupprio Ungrouped 3
104	* @groupprio setParam 4
105	* @groupprio getParam 5
106	* @groupdesc param
107	* A list of (hyper-)parameter keys this annotator can take. Users can set and get the
108	* parameter values through setters and getters, respectively.
109	*/
110	class LanguageDetectorDL(override val uid: String)
111	extends AnnotatorModel[LanguageDetectorDL]
112	with HasSimpleAnnotate[LanguageDetectorDL]
113	with WriteTensorflowModel {
114
115	def this() = this(Identifiable.randomUID("LANGUAGE_DETECTOR_DL"))	×
116
117	/** Alphabet used to feed the TensorFlow model for prediction
118	*
119	* @group param
120	*/
121	val alphabet: MapFeature[String, Int] = new MapFeature(this, "alphabet")	1✔
122
123	/** Language used to map prediction to ISO 639-1 language codes
124	*
125	* @group param
126	*/
127	val language: MapFeature[String, Int] = new MapFeature(this, "language")	1✔
128
129	/** The minimum threshold for the final result, otherwise it will be either `"unk"` or the value
130	* set in `thresholdLabel` (Default: `0.1f`). Value is between 0.0 to 1.0. Try to set this
131	* lower if your text is hard to predict
132	*
133	* @group param
134	*/
135	val threshold = new FloatParam(	1✔
136	this,
137	"threshold",	1✔
138	"The minimum threshold for the final result otherwise it will be either Unknown or the value set in thresholdLabel.")	1✔
139
140	/** Value for the classification, if confidence is less than `threshold` (Default: `"unk"`).
141	*
142	* @group param
143	*/
144	val thresholdLabel = new Param[String](	1✔
145	this,
146	"thresholdLabel",	1✔
147	"In case the score is less than threshold, what should be the label. Default is Unknown.")	1✔
148
149	/** Output average of sentences instead of one output per sentence (Default: `true`).
150	*
151	* @group param
152	*/
153	val coalesceSentences = new BooleanParam(	1✔
154	this,
155	"coalesceSentences",	1✔
156	"If sets to true the output of all sentences will be averaged to one output instead of one output per sentence. Default to true.")	1✔
157
158	/** ConfigProto from tensorflow, serialized into byte array. Get with
159	* config_proto.SerializeToString()
160	*
161	* @group param
162	*/
163	val configProtoBytes = new IntArrayParam(	1✔
164	this,
165	"configProtoBytes",	1✔
166	"ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()")	1✔
167
168	/** Languages the model was trained with.
169	*
170	* @group param
171	*/
172	val languages =
173	new StringArrayParam(this, "languages", "keep an internal copy of languages for Python")	1✔
174
175	/** @group setParam */
176	def setLanguage(value: Map[String, Int]): this.type = {
177	if (get(language).isEmpty)	×
178	set(this.language, value)	×
179	this
180	}
181
182	/** @group setParam */
183	def setAlphabet(value: Map[String, Int]): this.type = {
184	if (get(language).isEmpty)	×
185	set(alphabet, value)	×
186	this
187	}
188
189	/** @group setParam */
190	def setThreshold(threshold: Float): this.type = set(this.threshold, threshold)	×
191
192	/** @group setParam */
193	def setThresholdLabel(label: String): this.type = set(this.thresholdLabel, label)	×
194
195	/** @group setParam */
196	def setCoalesceSentences(value: Boolean): this.type = set(coalesceSentences, value)	×
197
198	/** @group setParam */
199	def setConfigProtoBytes(bytes: Array[Int]): LanguageDetectorDL.this.type =
200	set(this.configProtoBytes, bytes)	×
201
202	/** @group getParam */
203	def getLanguage: Array[String] = {
204	val langs = $$(language).keys.toArray	×
205	set(languages, langs)	×
206	langs
207	}
208
209	/** @group getParam */
210	def getThreshold: Float = $(this.threshold)	×
211
212	/** @group getParam */
213	def getThresholdLabel: String = $(this.thresholdLabel)	×
214
215	/** @group getParam */
216	def getCoalesceSentences: Boolean = $(coalesceSentences)	×
217
218	/** @group getParam */
219	def getConfigProtoBytes: Option[Array[Byte]] = get(this.configProtoBytes).map(_.map(_.toByte))	×
220
221	setDefault(	1✔
222	inputCols -> Array("document"),	1✔
223	outputCol -> "language",	1✔
224	threshold -> 0.1f,	1✔
225	thresholdLabel -> "unk",	1✔
226	coalesceSentences -> true)	1✔
227
228	private var _model: Option[Broadcast[TensorflowLD]] = None	1✔
229
230	/** @group getParam */
231	def getModelIfNotSet: TensorflowLD = _model.get.value	1✔
232
233	/** @group setParam */
234	def setModelIfNotSet(spark: SparkSession, tensorflow: TensorflowWrapper): this.type = {
235	if (_model.isEmpty) {	1✔
236
237	_model = Some(	1✔
238	spark.sparkContext.broadcast(	1✔
239	new TensorflowLD(	1✔
240	tensorflow,
241	configProtoBytes = getConfigProtoBytes,	1✔
242	ListMap($$(language).toSeq.sortBy(_._2): _*),	1✔
243	ListMap($$(alphabet).toSeq.sortBy(_._2): _*))))	1✔
244	}
245
246	this
247	}
248
249	/** Takes a document and annotations and produces new annotations of this annotator's annotation
250	* type
251	*
252	* @param annotations
253	* Annotations that correspond to inputAnnotationCols generated by previous annotators if any
254	* @return
255	* any number of annotations processed for every input annotation. Not necessary one to one
256	* relationship
257	*/
258	override def annotate(annotations: Seq[Annotation]): Seq[Annotation] = {
259	val sentences = SentenceSplit.unpack(annotations)	1✔
260	val nonEmptySentences = sentences.filter(_.content.nonEmpty)	1✔
261	if (nonEmptySentences.nonEmpty) {	1✔
262	getModelIfNotSet.predict(	1✔
263	nonEmptySentences,
264	$(threshold),	1✔
265	$(thresholdLabel),	1✔
266	$(coalesceSentences))	1✔
267	} else {
268	Seq.empty[Annotation]	×
269	}
270	}
271
272	/** Annotator reference id. Used to identify elements in metadata or to refer to this annotator
273	* type
274	*/
275	override val inputAnnotatorTypes: Array[String] = Array(AnnotatorType.DOCUMENT)	1✔
276	override val outputAnnotatorType: AnnotatorType = AnnotatorType.LANGUAGE	1✔
277
278	override def onWrite(path: String, spark: SparkSession): Unit = {
279	super.onWrite(path, spark)	1✔
280	writeTensorflowModelV2(	1✔
281	path,
282	spark,
283	getModelIfNotSet.tensorflow,	1✔
284	"_ld",	1✔
285	LanguageDetectorDL.tfFile,	1✔
286	configProtoBytes = getConfigProtoBytes)	1✔
287	}
288
289	}
290
291	trait ReadablePretrainedLanguageDetectorDLModel
292	extends ParamsAndFeaturesReadable[LanguageDetectorDL]
293	with HasPretrained[LanguageDetectorDL] {
294	override val defaultModelName: Some[String] = Some("ld_wiki_tatoeba_cnn_21")	1✔
295	override val defaultLang: String = "xx"	1✔
296
297	/** Java compliant-overrides */
298	override def pretrained(): LanguageDetectorDL = super.pretrained()	1✔
299
300	override def pretrained(name: String): LanguageDetectorDL = super.pretrained(name)	×
301
302	override def pretrained(name: String, lang: String): LanguageDetectorDL =
303	super.pretrained(name, lang)	×
304
305	override def pretrained(name: String, lang: String, remoteLoc: String): LanguageDetectorDL =
306	super.pretrained(name, lang, remoteLoc)	1✔
307	}
308
309	trait ReadLanguageDetectorDLTensorflowModel extends ReadTensorflowModel {
310	this: ParamsAndFeaturesReadable[LanguageDetectorDL] =>
311
312	override val tfFile: String = "ld_tensorflow"	1✔
313
314	def readTensorflow(instance: LanguageDetectorDL, path: String, spark: SparkSession): Unit = {
315
316	val tf = readTensorflowModel(path, spark, "_ld_tf")	1✔
317	instance.setModelIfNotSet(spark, tf)	1✔
318	// This allows for Python to access getLanguages function
319	val t = instance.language.get.toArray	1✔
320	val r = t(0).keys.toArray	1✔
321	instance.set(instance.languages, r)	1✔
322	}
323
324	addReader(readTensorflow)	1✔
325
326	def loadSavedModel(folder: String, spark: SparkSession): LanguageDetectorDL = {
327
328	val f = new File(folder)	×
329	val savedModel = new File(folder, "saved_model.pb")	×
330	require(f.exists, s"Folder $folder not found")	×
331	require(f.isDirectory, s"File $folder is not folder")	×
332	require(savedModel.exists(), s"savedModel file saved_model.pb not found in folder $folder")	×
333
334	val alphabetPath = new File(folder + "/assets", "alphabet.txt")	×
335	val languagePath = new File(folder + "/assets", "language.txt")	×
336
337	require(f.exists, s"Folder $folder not found")	×
338	require(f.isDirectory, s"File $folder is not folder")	×
339	require(alphabetPath.exists(), s"Alphabet file alphabet.txt not found in folder $folder")	×
340	require(languagePath.exists(), s"Language file language.txt not found in folder $folder")	×
341
342	val alphabetResource =
343	new ExternalResource(alphabetPath.getAbsolutePath, ReadAs.TEXT, Map("format" -> "text"))	×
344	val alphabets = ResourceHelper.parseLines(alphabetResource).zipWithIndex.toMap	×
345
346	val languageResource =
347	new ExternalResource(languagePath.getAbsolutePath, ReadAs.TEXT, Map("format" -> "text"))	×
348	val languages = ResourceHelper.parseLines(languageResource).zipWithIndex.toMap	×
349
350	val (wrapper, _) =
351	TensorflowWrapper.read(folder, zipped = false, useBundle = true, tags = Array("serve"))	×
352
353	new LanguageDetectorDL()
354	.setAlphabet(alphabets)
355	.setLanguage(languages)
356	.setModelIfNotSet(spark, wrapper)	×
357	}
358	}
359
360	/** This is the companion object of [[LanguageDetectorDL]]. Please refer to that class for the
361	* documentation.
362	*/
363	object LanguageDetectorDL
364	extends ReadablePretrainedLanguageDetectorDLModel
365	with ReadLanguageDetectorDLTensorflowModel

JohnSnowLabs / spark-nlp / 2114953157

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous