I'm trying to convert a tensorflow model to CoreML model.
Here is what I do
import tensorflow as tf import coremltools as ct
# copy paste the labels class_labels = [ 'label 1', ['label 2'] .. ['label n']]
model = tf.keras.models.load_model('my_tf_model')
config = model.get_config() # check out the model config
print(config)
#convert to CoreMl model mlmodel = ct.convert(model
, convert_to="mlprogram",
inputs=[ct.ImageType(
name="sequential_1_input",
shape=(1, 256, 256, 3),
# scale=1.0 / 255.0,
bias=[-10,-10,-10],
color_layout="RGB"
)],
# outputs=[ct.ImageType()],
source="tensorflow",
classifier_config=ct.ClassifierConfig(class_labels)#set the labels
) shape=(1, 256, 256, 3) -- 1(I don't know what it is)
256,256(model width and height), 3 again - I don't know what it is
bias=[-100,-100,-100] - I'm using this to normalize the probability, else I get -0.2, -2.0. With this I'm actually getting something like 20%, 50%
Than I save the model
mlmodel.save("MyModel.mlpackage")
The model is saved with the name MyModel.mlpackage as expected
Then in xcode I'm using the model
let config = MLModelConfiguration()
let imageClassifier = try MyModel(modelconfig: config)
let imageClassifierModel = imageClassifier.model
let result = try imageClassifier.prediction(sequential_1_input: buffer!)
// get the label and probability
let className: String = result.classLabel
let confidence: Double = result.classLabel_probs[result.classLabel] ?? 0.0
Again, all good.
The problem is that the prediction is wrong, each time. If I make the same prediction with the same image in Python works with high probability (100% most of the time)
So Why? Is coreML that bad? Am I doing something wrong?
Should I switch to tensorlite? Is there any other option for ios?
As far as I understood the CoreML it supposed to normalize the image pixel, and it looks like it doesn't, This is the only thing that I can think of, but I have no idea how to do this in CoreML.
update, I managed to figure out a problem. The autotuneAUTOTUNE =
tf.data.AUTOTUNE
train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)
normalization_layer = layers.Rescaling(1./255)
normalized_ds = train_ds.map(lambda x, y: (normalization_layer(x), y))
image_batch, labels_batch = next(iter(normalized_ds))
first_image = image_batch[0]
# Notice the pixel values are now in `[0,1]`.
print(np.min(first_image), np.max(first_image))
So I'm guessing is the normalization. I tried to use the scaling to reproduce this in coremltools but it does not work.
Once I commented out this code and went to xcode and drag and drop images in the preview, it works as expected, but not in code.
So I'm guessing the issue is within the buffer, where I'm converting the image to what coremltools need
func buffer(with size:CGSize) -> CVPixelBuffer? {
if let image = self.cgImage {
let frameSize = size
var pixelBuffer:CVPixelBuffer? = nil
let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(frameSize.width), Int(frameSize.height), kCVPixelFormatType_24ARGB , nil, &pixelBuffer)
if status != kCVReturnSuccess {
return nil
}
CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags.init(rawValue: 0))
let data = CVPixelBufferGetBaseAddress(pixelBuffer!)
let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
let bitmapInfo = CGBitmapInfo(rawValue: CGBitmapInfo.byteOrder32Little.rawValue | CGImageAlphaInfo.premultipliedFirst.rawValue)
let context = CGContext(data: data, width: Int(frameSize.width), height: Int(frameSize.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: bitmapInfo.rawValue)
context?.draw(image, in: CGRect(x: 0, y: 0, width: image.width, height: image.height))
CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
return pixelBuffer
}else{
return nil
}
}
And I think is because of the way the buffer is created
kCVPixelFormatType_24RGBA
If I print the mlmodel I get
input {
name: "sequential_1_input"
type {
imageType {
width: 64
height: 64
colorSpace: RGB
}
}
}
output {....
So this means the input must be rgb and not rgba or argb or anything But it doesn't say if is 24 or 32
If I drag and drop images in the model previwer works perfect with the code commented(see up)
And I might be missing something else, I just couldn't figure out what.
I finally make it work
So the conversion was inaccurate because of a library I was using, it was a warning asking me to downgrade, and initially, I thought that it works, why should I use an older version? But then I said, let me give to caesar what belongs to caesar. So now it actually works!!!
Now for CoreML:
Thank you Jeshua Lacock for your comment