5210642866

pending completion

Build # 5210642866

Build Type

Pull #13848

github

Committed by

web-flow

Commit Message

Merge d7636a247 into ddf39b36d

Pull Request Pull Request #13848: release/444-release-candidate

Run Details

3 of 3 new or added lines in 3 files covered. (100.0%)

8644 of 13124 relevant lines covered (65.86%)

0.66 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

0.0

/src/main/scala/com/johnsnowlabs/nlp/annotators/audio/HubertForCTC.scala

/*
 * Copyright 2017-2022 John Snow Labs
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package com.johnsnowlabs.nlp.annotators.audio

import com.johnsnowlabs.ml.tensorflow.{ReadTensorflowModel, TensorflowWrapper}
import com.johnsnowlabs.ml.util.LoadExternalModel.{
  loadJsonStringAsset,
  modelSanityCheck,
  notSupportedEngineError
}
import com.johnsnowlabs.ml.util.ModelEngine
import com.johnsnowlabs.nlp._
import com.johnsnowlabs.nlp.annotators.audio.feature_extractor.Preprocessor
import org.apache.spark.ml.util.Identifiable
import org.apache.spark.sql.SparkSession
import org.json4s._
import org.json4s.jackson.JsonMethods._

/** Hubert Model with a language modeling head on top for Connectionist Temporal Classification
  * (CTC). Hubert was proposed in HuBERT: Self-Supervised Speech Representation Learning by Masked
  * Prediction of Hidden Units by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal
  * Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed.
  *
  * The annotator takes audio files and transcribes it as text. The audio needs to be provided
  * pre-processed an array of floats.
  *
  * Note that this annotator is currently not supported on Apple Silicon processors such as the
  * M1/M2 (Apple Silicon). This is due to the processor not supporting instructions for XLA.
  *
  * Pretrained models can be loaded with `pretrained` of the companion object:
  * {{{
  * val speechToText = HubertForCTC.pretrained()
  *   .setInputCols("audio_assembler")
  *   .setOutputCol("text")
  * }}}
  * The default model is `"asr_hubert_large_ls960"`, if no name is provided.
  *
  * For available pretrained models please see the [[https://sparknlp.org/models Models Hub]].
  *
  * To see which models are compatible and how to import them see
  * [[https://github.com/JohnSnowLabs/spark-nlp/discussions/5669]] and to see more extended
  * examples, see
  * [[https://github.com/JohnSnowLabs/spark-nlp/blob/master/src/test/scala/com/johnsnowlabs/nlp/annotators/audio/HubertForCTCTestSpec.scala HubertForCTCTestSpec]].
  *
  * '''References:'''
  *
  * [[https://arxiv.org/abs/2106.07447 HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units]]
  *
  * '''Paper Abstract:'''
  *
  * ''Self-supervised approaches for speech representation learning are challenged by three unique
  * problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon
  * of input sound units during the pre-training phase, and (3) sound units have variable lengths
  * with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit
  * BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an
  * offline clustering step to provide aligned target labels for a BERT-like prediction loss. A
  * key ingredient of our approach is applying the prediction loss over the masked regions only,
  * which forces the model to learn a combined acoustic and language model over the continuous
  * inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather
  * than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means
  * teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either
  * matches or improves upon the state-of-the-art wav2vec 2.0 performance on the Librispeech
  * (960h) and Libri-light (60,000h) benchmarks with 10min, 1h, 10h, 100h, and 960h fine-tuning
  * subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER reduction on
  * the more challenging dev-other and test-other evaluation subsets.''
  *
  * ==Example==
  * {{{
  * import spark.implicits._
  * import com.johnsnowlabs.nlp.base._
  * import com.johnsnowlabs.nlp.annotators._
  * import com.johnsnowlabs.nlp.annotators.audio.HubertForCTC
  * import org.apache.spark.ml.Pipeline
  *
  * val audioAssembler: AudioAssembler = new AudioAssembler()
  *   .setInputCol("audio_content")
  *   .setOutputCol("audio_assembler")
  *
  * val speechToText: HubertForCTC = HubertForCTC
  *   .pretrained()
  *   .setInputCols("audio_assembler")
  *   .setOutputCol("text")
  *
  * val pipeline: Pipeline = new Pipeline().setStages(Array(audioAssembler, speechToText))
  *
  * val bufferedSource =
  *   scala.io.Source.fromFile("src/test/resources/audio/csv/audio_floats.csv")
  *
  * val rawFloats = bufferedSource
  *   .getLines()
  *   .map(_.split(",").head.trim.toFloat)
  *   .toArray
  * bufferedSource.close
  *
  * val processedAudioFloats = Seq(rawFloats).toDF("audio_content")
  *
  * val result = pipeline.fit(processedAudioFloats).transform(processedAudioFloats)
  * result.select("text.result").show(truncate = false)
  * +------------------------------------------------------------------------------------------+
  * |result                                                                                    |
  * +------------------------------------------------------------------------------------------+
  * |[MISTER QUILTER IS THE APOSTLE OF THE MIDLE CLASES AND WE ARE GLAD TO WELCOME HIS GOSPEL ]|
  * +------------------------------------------------------------------------------------------+
  * }}}
  *
  * @param uid
  *   required uid for storing annotator to disk
  * @groupname anno Annotator types
  * @groupdesc anno
  *   Required input and expected output annotator types
  * @groupname Ungrouped Members
  * @groupname param Parameters
  * @groupname setParam Parameter setters
  * @groupname getParam Parameter getters
  * @groupname Ungrouped Members
  * @groupprio param  1
  * @groupprio anno  2
  * @groupprio Ungrouped 3
  * @groupprio setParam  4
  * @groupprio getParam  5
  * @groupdesc param
  *   A list of (hyper-)parameter keys this annotator can take. Users can set and get the
  *   parameter values through setters and getters, respectively.
  */
class HubertForCTC(override val uid: String) extends Wav2Vec2ForCTC(uid) {

  /** Annotator reference id. Used to identify elements in metadata or to refer to this annotator
    * type
    */
  def this() = this(Identifiable.randomUID("HubertForCTC"))

  override def onWrite(path: String, spark: SparkSession): Unit = {
    super.onWrite(path, spark)
    writeTensorflowModelV2(
      path,
      spark,
      getModelIfNotSet.tensorflowWrapper,
      "_hubert_ctc",
      HubertForCTC.tfFile,
      configProtoBytes = getConfigProtoBytes)
  }

}

trait ReadablePretrainedHubertForAudioModel
    extends ParamsAndFeaturesReadable[HubertForCTC]
    with HasPretrained[HubertForCTC] {
  override val defaultModelName: Some[String] = Some("asr_hubert_large_ls960")

  /** Java compliant-overrides */
  override def pretrained(): HubertForCTC = super.pretrained()

  override def pretrained(name: String): HubertForCTC = super.pretrained(name)

  override def pretrained(name: String, lang: String): HubertForCTC =
    super.pretrained(name, lang)

  override def pretrained(name: String, lang: String, remoteLoc: String): HubertForCTC =
    super.pretrained(name, lang, remoteLoc)
}

trait ReadHubertForAudioDLModel extends ReadTensorflowModel {
  this: ParamsAndFeaturesReadable[HubertForCTC] =>

  override val tfFile: String = "hubert_ctc_tensorflow"

  def readTensorflow(instance: HubertForCTC, path: String, spark: SparkSession): Unit = {

    val tf = readTensorflowModel(path, spark, "_hubert_ctc_tf", initAllTables = false)
    instance.setModelIfNotSet(spark, tf)
  }

  addReader(readTensorflow)

  def loadSavedModel(modelPath: String, spark: SparkSession): HubertForCTC = {

    val (localModelPath, detectedEngine) = modelSanityCheck(modelPath)

    val vocabJsonContent = loadJsonStringAsset(localModelPath, "vocab.json")
    val vocabJsonMap =
      parse(vocabJsonContent, useBigIntForLong = true).values
        .asInstanceOf[Map[String, BigInt]]

    val preprocessorConfigJsonContent =
      loadJsonStringAsset(localModelPath, "preprocessor_config.json")
    val preprocessorConfig =
      Preprocessor.loadPreprocessorConfig(preprocessorConfigJsonContent)

    /*Universal parameters for all engines*/
    val annotatorModel = new HubertForCTC()
      .setVocabulary(vocabJsonMap)
      .setDoNormalize(preprocessorConfig.do_normalize)
      .setFeatureSize(preprocessorConfig.feature_size)
      .setPaddingSide(preprocessorConfig.padding_side)
      .setPaddingValue(preprocessorConfig.padding_value)
      .setReturnAttentionMask(preprocessorConfig.return_attention_mask)
      .setSamplingRate(preprocessorConfig.sampling_rate)

    annotatorModel.set(annotatorModel.engine, detectedEngine)

    detectedEngine match {
      case ModelEngine.tensorflow =>
        val (wrapper, signatures) =
          TensorflowWrapper.read(localModelPath, zipped = false, useBundle = true)

        val _signatures = signatures match {
          case Some(s) => s
          case None => throw new Exception("Cannot load signature definitions from model!")
        }

        /** the order of setSignatures is important if we use getSignatures inside
          * setModelIfNotSet
          */
        annotatorModel
          .setSignatures(_signatures)
          .setModelIfNotSet(spark, wrapper)

      case _ =>
        throw new Exception(notSupportedEngineError)
    }

    annotatorModel
  }
}

/** This is the companion object of [[HubertForCTC]]. Please refer to that class for the
  * documentation.
  */
object HubertForCTC extends ReadablePretrainedHubertForAudioModel with ReadHubertForAudioDLModel

1	/*
2	* Copyright 2017-2022 John Snow Labs
3	*
4	* Licensed under the Apache License, Version 2.0 (the "License");
5	* you may not use this file except in compliance with the License.
6	* You may obtain a copy of the License at
7	*
8	* http://www.apache.org/licenses/LICENSE-2.0
9	*
10	* Unless required by applicable law or agreed to in writing, software
11	* distributed under the License is distributed on an "AS IS" BASIS,
12	* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13	* See the License for the specific language governing permissions and
14	* limitations under the License.
15	*/
16
17	package com.johnsnowlabs.nlp.annotators.audio
18
19	import com.johnsnowlabs.ml.tensorflow.{ReadTensorflowModel, TensorflowWrapper}
20	import com.johnsnowlabs.ml.util.LoadExternalModel.{
21	loadJsonStringAsset,
22	modelSanityCheck,
23	notSupportedEngineError
24	}
25	import com.johnsnowlabs.ml.util.ModelEngine
26	import com.johnsnowlabs.nlp._
27	import com.johnsnowlabs.nlp.annotators.audio.feature_extractor.Preprocessor
28	import org.apache.spark.ml.util.Identifiable
29	import org.apache.spark.sql.SparkSession
30	import org.json4s._
31	import org.json4s.jackson.JsonMethods._
32
33	/** Hubert Model with a language modeling head on top for Connectionist Temporal Classification
34	* (CTC). Hubert was proposed in HuBERT: Self-Supervised Speech Representation Learning by Masked
35	* Prediction of Hidden Units by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal
36	* Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed.
37	*
38	* The annotator takes audio files and transcribes it as text. The audio needs to be provided
39	* pre-processed an array of floats.
40	*
41	* Note that this annotator is currently not supported on Apple Silicon processors such as the
42	* M1/M2 (Apple Silicon). This is due to the processor not supporting instructions for XLA.
43	*
44	* Pretrained models can be loaded with `pretrained` of the companion object:
45	* {{{
46	* val speechToText = HubertForCTC.pretrained()
47	* .setInputCols("audio_assembler")
48	* .setOutputCol("text")
49	* }}}
50	* The default model is `"asr_hubert_large_ls960"`, if no name is provided.
51	*
52	* For available pretrained models please see the [[https://sparknlp.org/models Models Hub]].
53	*
54	* To see which models are compatible and how to import them see
55	* [[https://github.com/JohnSnowLabs/spark-nlp/discussions/5669]] and to see more extended
56	* examples, see
57	* [[https://github.com/JohnSnowLabs/spark-nlp/blob/master/src/test/scala/com/johnsnowlabs/nlp/annotators/audio/HubertForCTCTestSpec.scala HubertForCTCTestSpec]].
58	*
59	* '''References:'''
60	*
61	* [[https://arxiv.org/abs/2106.07447 HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units]]
62	*
63	* '''Paper Abstract:'''
64	*
65	* ''Self-supervised approaches for speech representation learning are challenged by three unique
66	* problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon
67	* of input sound units during the pre-training phase, and (3) sound units have variable lengths
68	* with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit
69	* BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an
70	* offline clustering step to provide aligned target labels for a BERT-like prediction loss. A
71	* key ingredient of our approach is applying the prediction loss over the masked regions only,
72	* which forces the model to learn a combined acoustic and language model over the continuous
73	* inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather
74	* than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means
75	* teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either
76	* matches or improves upon the state-of-the-art wav2vec 2.0 performance on the Librispeech
77	* (960h) and Libri-light (60,000h) benchmarks with 10min, 1h, 10h, 100h, and 960h fine-tuning
78	* subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER reduction on
79	* the more challenging dev-other and test-other evaluation subsets.''
80	*
81	* ==Example==
82	* {{{
83	* import spark.implicits._
84	* import com.johnsnowlabs.nlp.base._
85	* import com.johnsnowlabs.nlp.annotators._
86	* import com.johnsnowlabs.nlp.annotators.audio.HubertForCTC
87	* import org.apache.spark.ml.Pipeline
88	*
89	* val audioAssembler: AudioAssembler = new AudioAssembler()
90	* .setInputCol("audio_content")
91	* .setOutputCol("audio_assembler")
92	*
93	* val speechToText: HubertForCTC = HubertForCTC
94	* .pretrained()
95	* .setInputCols("audio_assembler")
96	* .setOutputCol("text")
97	*
98	* val pipeline: Pipeline = new Pipeline().setStages(Array(audioAssembler, speechToText))
99	*
100	* val bufferedSource =
101	* scala.io.Source.fromFile("src/test/resources/audio/csv/audio_floats.csv")
102	*
103	* val rawFloats = bufferedSource
104	* .getLines()
105	* .map(_.split(",").head.trim.toFloat)
106	* .toArray
107	* bufferedSource.close
108	*
109	* val processedAudioFloats = Seq(rawFloats).toDF("audio_content")
110	*
111	* val result = pipeline.fit(processedAudioFloats).transform(processedAudioFloats)
112	* result.select("text.result").show(truncate = false)
113	* +------------------------------------------------------------------------------------------+
114	* \|result \|
115	* +------------------------------------------------------------------------------------------+
116	* \|[MISTER QUILTER IS THE APOSTLE OF THE MIDLE CLASES AND WE ARE GLAD TO WELCOME HIS GOSPEL ]\|
117	* +------------------------------------------------------------------------------------------+
118	* }}}
119	*
120	* @param uid
121	* required uid for storing annotator to disk
122	* @groupname anno Annotator types
123	* @groupdesc anno
124	* Required input and expected output annotator types
125	* @groupname Ungrouped Members
126	* @groupname param Parameters
127	* @groupname setParam Parameter setters
128	* @groupname getParam Parameter getters
129	* @groupname Ungrouped Members
130	* @groupprio param 1
131	* @groupprio anno 2
132	* @groupprio Ungrouped 3
133	* @groupprio setParam 4
134	* @groupprio getParam 5
135	* @groupdesc param
136	* A list of (hyper-)parameter keys this annotator can take. Users can set and get the
137	* parameter values through setters and getters, respectively.
138	*/
139	class HubertForCTC(override val uid: String) extends Wav2Vec2ForCTC(uid) {
140
141	/** Annotator reference id. Used to identify elements in metadata or to refer to this annotator
142	* type
143	*/
144	def this() = this(Identifiable.randomUID("HubertForCTC"))	×
145
146	override def onWrite(path: String, spark: SparkSession): Unit = {
147	super.onWrite(path, spark)	×
148	writeTensorflowModelV2(	×
149	path,
150	spark,
151	getModelIfNotSet.tensorflowWrapper,	×
152	"_hubert_ctc",	×
153	HubertForCTC.tfFile,	×
154	configProtoBytes = getConfigProtoBytes)	×
155	}
156
157	}
158
159	trait ReadablePretrainedHubertForAudioModel
160	extends ParamsAndFeaturesReadable[HubertForCTC]
161	with HasPretrained[HubertForCTC] {
162	override val defaultModelName: Some[String] = Some("asr_hubert_large_ls960")	×
163
164	/** Java compliant-overrides */
165	override def pretrained(): HubertForCTC = super.pretrained()	×
166
167	override def pretrained(name: String): HubertForCTC = super.pretrained(name)	×
168
169	override def pretrained(name: String, lang: String): HubertForCTC =
170	super.pretrained(name, lang)	×
171
172	override def pretrained(name: String, lang: String, remoteLoc: String): HubertForCTC =
173	super.pretrained(name, lang, remoteLoc)	×
174	}
175
176	trait ReadHubertForAudioDLModel extends ReadTensorflowModel {
177	this: ParamsAndFeaturesReadable[HubertForCTC] =>
178
179	override val tfFile: String = "hubert_ctc_tensorflow"	×
180
181	def readTensorflow(instance: HubertForCTC, path: String, spark: SparkSession): Unit = {
182
183	val tf = readTensorflowModel(path, spark, "_hubert_ctc_tf", initAllTables = false)	×
184	instance.setModelIfNotSet(spark, tf)	×
185	}
186
187	addReader(readTensorflow)	×
188
189	def loadSavedModel(modelPath: String, spark: SparkSession): HubertForCTC = {
190
191	val (localModelPath, detectedEngine) = modelSanityCheck(modelPath)	×
192
193	val vocabJsonContent = loadJsonStringAsset(localModelPath, "vocab.json")	×
194	val vocabJsonMap =
195	parse(vocabJsonContent, useBigIntForLong = true).values	×
196	.asInstanceOf[Map[String, BigInt]]	×
197
198	val preprocessorConfigJsonContent =
199	loadJsonStringAsset(localModelPath, "preprocessor_config.json")	×
200	val preprocessorConfig =
201	Preprocessor.loadPreprocessorConfig(preprocessorConfigJsonContent)	×
202
203	/Universal parameters for all engines/
204	val annotatorModel = new HubertForCTC()
205	.setVocabulary(vocabJsonMap)
206	.setDoNormalize(preprocessorConfig.do_normalize)	×
207	.setFeatureSize(preprocessorConfig.feature_size)	×
208	.setPaddingSide(preprocessorConfig.padding_side)	×
209	.setPaddingValue(preprocessorConfig.padding_value)	×
210	.setReturnAttentionMask(preprocessorConfig.return_attention_mask)	×
211	.setSamplingRate(preprocessorConfig.sampling_rate)	×
212
213	annotatorModel.set(annotatorModel.engine, detectedEngine)	×
214
215	detectedEngine match {
216	case ModelEngine.tensorflow =>
217	val (wrapper, signatures) =	×
218	TensorflowWrapper.read(localModelPath, zipped = false, useBundle = true)
219
220	val _signatures = signatures match {
221	case Some(s) => s
222	case None => throw new Exception("Cannot load signature definitions from model!")	×
223	}
224
225	/** the order of setSignatures is important if we use getSignatures inside
226	* setModelIfNotSet
227	*/
228	annotatorModel
229	.setSignatures(_signatures)
230	.setModelIfNotSet(spark, wrapper)	×
231
232	case _ =>
233	throw new Exception(notSupportedEngineError)	×
234	}
235
236	annotatorModel
237	}
238	}
239
240	/** This is the companion object of [[HubertForCTC]]. Please refer to that class for the
241	* documentation.
242	*/
243	object HubertForCTC extends ReadablePretrainedHubertForAudioModel with ReadHubertForAudioDLModel

JohnSnowLabs / spark-nlp / 5210642866

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous