iOS 14 ARKit CapturedImage JPEG Conversion HEIC Warning

Question

TL;DR: Something has changed with ARKit capturedImage in iOS14, and we couldn't make it fit into our flow.

Built with Xcode 12 GM, tested on iPhone 6S 14.0.

We have implemented a flow where we transform ARFrame to Data, which works perfectly for devices that are not upgraded to iOS14 yet.

We are getting the ciImage like:

extension ARFrame {
    var ciImage: CIImage { CIImage(cvPixelBuffer: capturedImage) }
}

which we feed in an ImageConverter:

public class ImageConverter {
    lazy var imageContext = CIContext(mtlDevice: MTLCreateSystemDefaultDevice()!)
    lazy var colorSpace = CGColorSpace(name: CGColorSpace.sRGB)

    public func jpgData(for image: CIImage) -> Data {
        guard let colorSpace = colorSpace else { return Data() }

        if let jpegRepresentation = imageContext.jpegRepresentation(of: image,
                                                                    colorSpace: image.colorSpace ?? colorSpace) {
            return jpegRepresentation
        } else {
            return Data()
        }
    }
}

So, the problem is, when we call .jpegRepresentation, it gives a warning:

findWriterForType:128: unsupported file format 'public.heic'

And although the jpegRepresentation is not nil, it's not working for our server, which expects a jpeg data (which still works for iOS13.7 or below devices).

What we tried so far:

if let heifRepresentation = imageContext.heifRepresentation(of: image,
                                                            format: .RGBA8,
                                                            colorSpace: image.colorSpace ?? colorSpace) {
    return heifRepresentation
}

returns nil, and throws the same error with .jpegRepresentation. Also tried other formats, to no avail.

Answer 1

The problem lies within the created JPEG data format with .jpegRepresentation.

Here's the difference between iPhone XS - iOS13 vs iPhone 6S - iOS14:

Terminal Output:

➜ file testiOS13.jpeg

testiOS13.jpeg: JPEG image data, JFIF standard 1.01, aspect ratio, density 72x72, segment length 16, Exif Standard: [TIFF image data, big-endian, direntries=1], baseline, precision 8, 600x800, components 3

➜ file testiOS14.jpeg

testiOS14.jpeg: JPEG image data, Exif standard: [TIFF image data, big-endian, direntries=5, xresolution=74, yresolution=82, resolutionunit=2], baseline, precision 8, 412x733, components 3

JFIF Standard 1.01 is missing on iOS 14 ARKit pictures.

With a hoop between CGImage and Data (and an availability check in case something goes amiss with iOS 13.7 or below):

public func jpgData(for image: CIImage, with size: CGSize) -> Data {
    guard #available(iOS 14.0, *) else { return jpgData(for: image) }

    guard let cgImage = imageContext.createCGImage(image, from: CGRect(origin: .zero, size: size)) else { return jpgData(for: image) }

    let data = NSMutableData()

    guard let imageDestination = CGImageDestinationCreateWithData(data, AVFileType.jpg as CFString, 1, nil) else { return jpgData(for: image) }
    CGImageDestinationAddImage(imageDestination, cgImage, nil)

    guard CGImageDestinationFinalize(imageDestination) else { return jpgData(for: image) }

    return data as Data
}

private func jpgData(for image: CIImage) -> Data {
    guard let colorSpace = colorSpace else { return Data() }

    return imageContext.jpegRepresentation(of: image, colorSpace: colorSpace) ?? Data()
}

With the latest change:

Terminal Output:

➜ file testiOS14_New.jpeg

testiOS14_New.jpeg: JPEG image data, JFIF standard 1.01, aspect ratio, density 72x72, segment length 16, Exif Standard: [TIFF image data, big-endian, direntries=1], baseline, precision 8, 413x734, components 3