I have been trying to implement the Simple Audio Recognition Tensorflow sample in iOS using the Firebase's ML kit. I have successfully trained the model and converted it into a TFlite file. The model takes the Audio(wav) file path as input([String]) and gives the predictions as output(float32). My iOS code is fairly simple
func initMLModel(){
/*Initializing local TFLite model*/
guard let modelPath = Bundle.main.path(forResource: "converted_model", ofType: "tflite") else {
return
}
let myLocalModel = LocalModelSource.init(modelName: "My", path: modelPath)
let registrationSuccessful = ModelManager.modelManager().register(myLocalModel)
let options = ModelOptions(cloudModelName: nil, localModelName: "My")
let interpreter = ModelInterpreter.modelInterpreter(options: options)
let ioOptions = ModelInputOutputOptions()
do {
try ioOptions.setInputFormat(index: 0, type: .unknown, dimensions: []) /*input is string path. Since string is not defined, setting it as unknown.*/
try ioOptions.setOutputFormat(index: 0, type: .float32, dimensions: [1,38]) /* output is 1 of 38 labelled classes*/
} catch let error as NSError {
print("Failed to set IO \(error.debugDescription)")
}
let inputs = ModelInputs()
var audioData = Data()
let audiopath = Bundle.main.path(forResource: "audio", ofType: "wav")
do {
audioData = try Data.init(contentsOf: URL.init(fileURLWithPath: audiopath!))
//try inputs.addInput(audioData) /*If the input type is direct audio data*/
try inputs.addInput([audiopath])
} catch let error as NSError {
print("Cannot get audio file data \(error.debugDescription)")
return
}
interpreter.run(inputs: inputs, options: ioOptions) { (outputs, error) in
if error != nil {
print("Error running the model \(error.debugDescription)")
return
}
do {
let output = try outputs!.output(index: 0) as? [[NSNumber]]
let probabilities = output?[0]
guard let labelsPath = Bundle.main.path(forResource: "conv_labels", ofType: "txt") else { return }
let fileContents = try? String.init(contentsOf: URL.init(fileURLWithPath: labelsPath))
guard let labels = fileContents?.components(separatedBy: "\n") else {return}
for i in 0 ..< labels.count {
if let probability = probabilities?[i] {
print("\(labels[i]) : \(probability)")
}
}
}catch let error as NSError {
print("Error in parsing the Output \(error.debugDescription)")
return
}
}
}
But when i run this i get the following error output Failed to create a TFLite interpreter for the given model. The Complete Log of the sample app is as below
2019-01-07 18:22:31.447917+0530 sample_core_ML[67500:3515789] - <AppMeasurement>[I-ACS036002] Analytics screen reporting is enabled. Call +[FIRAnalytics setScreenName:setScreenClass:] to set the screen name or override the default screen class name. To disable screen reporting, set the flag FirebaseScreenReportingEnabled to NO (boolean) in the Info.plist
2019-01-07 18:22:33.354449+0530 sample_core_ML[67500:3515686] libMobileGestalt MobileGestalt.c:890: MGIsDeviceOneOfType is not supported on this platform.
2019-01-07 18:22:34.789665+0530 sample_core_ML[67500:3515812] 5.15.0 - [Firebase/Analytics][I-ACS023007] Analytics v.50400000 started
2019-01-07 18:22:34.790814+0530 sample_core_ML[67500:3515812] 5.15.0 - [Firebase/Analytics][I-ACS023008] To enable debug logging set the following application argument: -FIRAnalyticsDebugEnabled (see )
2019-01-07 18:22:35.542993+0530 sample_core_ML[67500:3515823] [BoringSSL] nw_protocol_boringssl_get_output_frames(1301) [C1.1:2][0x7f9db0701d70] get output frames failed, state 8196
2019-01-07 18:22:35.543205+0530 sample_core_ML[67500:3515823] [BoringSSL] nw_protocol_boringssl_get_output_frames(1301) [C1.1:2][0x7f9db0701d70] get output frames failed, state 8196
2019-01-07 18:22:35.543923+0530 sample_core_ML[67500:3515823] TIC Read Status [1:0x0]: 1:57
2019-01-07 18:22:35.544070+0530 sample_core_ML[67500:3515823] TIC Read Status [1:0x0]: 1:57
2019-01-07 18:22:39.981492+0530 sample_core_ML[67500:3515823] 5.15.0 - [Firebase/MLKit][I-MLK002000] ModelInterpreterErrorReporter: Didn't find custom op for name 'DecodeWav' with version 1
2019-01-07 18:22:39.981686+0530 sample_core_ML[67500:3515823] 5.15.0 - [Firebase/MLKit][I-MLK002000] ModelInterpreterErrorReporter: Registration failed.
Failed to set IO Error Domain=com.firebase.ml Code=3 "input format 0 has invalid nil or empty dimensions." UserInfo={NSLocalizedDescription=input format 0 has invalid nil or empty dimensions.}
2019-01-07 18:22:40.604961+0530 sample_core_ML[67500:3515812] 5.15.0 - [Firebase/MLKit][I-MLK002000] ModelInterpreterErrorReporter: Didn't find custom op for name 'DecodeWav' with version 1
2019-01-07 18:22:40.605199+0530 sample_core_ML[67500:3515812] 5.15.0 - [Firebase/MLKit][I-MLK002000] ModelInterpreterErrorReporter: Registration failed.
Error running the model Optional(Error Domain=com.firebase.ml Code=2 "Failed to create a TFLite interpreter for the given model (/Users/minimaci73/Library/Developer/CoreSimulator/Devices/7FE413C1-3820-496A-B0CE-033BE2F3212A/data/Containers/Bundle/Application/868CB2FE-77D8-4B1F-8853-C2E17ECA63F2/sample_core_ML.app/converted_model.tflite)." UserInfo={NSLocalizedDescription=Failed to create a TFLite interpreter for the given model (/Users/minimaci73/Library/Developer/CoreSimulator/Devices/7FE413C1-3820-496A-B0CE-033BE2F3212A/data/Containers/Bundle/Application/868CB2FE-77D8-4B1F-8853-C2E17ECA63F2/sample_core_ML.app/converted_model.tflite).})
When looked at this line Didn't find custom op for name 'DecodeWav' I looked up on the custom supported ops and found that Tensorflow already supports this in the audio_ops.cc by default.
Details
My Tensorflow Version : 1.12.0
Environment : Conda
OS Version : Mac OSX Mojave 10.14.2
Deployment target : ios 12.0
Installation type : Pod Installation (pod 'Firebase/MLModelInterpreter')
But i ran my training model first in v1.9.0. Then updated the Tensorflow to latest v1.12.0 to run the TFLite Convertor. Both are master branch.
My TFLite Convertor code Python
import tensorflow as tf
graph_def_file = "my_frozen_graph.pb"
input_arrays = ["wav_data"]
output_arrays = ["labels_softmax"]
input_shape = {"wav_data" : [1,99,40,1]}
converter = tf.contrib.lite.TFLiteConverter.from_frozen_graph(
graph_def_file, input_arrays, output_arrays, input_shape)
converter.allow_custom_ops = True
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)
I posted this same question in the firebase quickstart iOS repository, And i got the following response DecodeWav op is never supported by TensorFlowLite. So at present Tensorflow Lite does not support audio processing even-though Tensorflow itself supports audio processing.