ZIPFoundation: Issue with writing large PNG to archive via data provider

UPDATE: I can reproduce this by setting the PNG size to be over an arbitrary value (i.e. 700 x 700 pt fails). Under the arbitrary value writes and reads fine. I'm not sure of exactly where that line is.

I'm using a zip archive as my document file format. I'm seeing unexpected results when trying to read in a PNG preview from multiple archives for a file browser page.

The document URLs are queried on the background and then a file data object is created. Once the query completes an UI update is called on the main thread and the file data objects act as a data provider to the collection view.

The PNG is serialized to Data like this:

let imageData = UIImagePNGRepresentation(image)

When the data is read the relevant entries are extracted into memory and then deserialized into their respective objects.

What I'm seeing is that the document thumbnails are only displayed intermittently. Is it possible that ZIPFoundation is not thread safe for background operations?

I've mocked this up in a simpler test project and it works fine but I'm not reading multiple archives and I'm not calling this in the background. Am I doing something wrong (code below)?

Is the consumer closure using its own thread (and maybe I'm returning data before it can complete)?

do {
    let decoder = JSONDecoder()
    let archive = Archive(url: URL, accessMode: .read)

    // Metadata
    if let metadataData = try archive?.readData(named: Document.MetadataFilename) {
        self.metadata = try decoder.decode(Metadata.self, from: metadataData)
    } else {
        logDebug(self, "metadata not read")
    }

    // Preview
    if let previewData = try archive?.readData(named: Document.PreviewFilename) {
        if let image = UIImage(data: previewData) {
            self.image = image
            return
        }
    } else {
        logDebug(self, "image not read")
    }

} catch {
    logError(self, "Loading of FileWrapper failed with error: \(error.localizedDescription)")
}

// Failure fall through
// Mark this as failed by using the x image
self.image = UIImage(named: "unavailable")
}

My extension to Archive for convenience:

/// Associates optional data with entry name
struct NamedData {
    let name : String
    let data : Data?
}

// MARK: - Private
extension Archive {

    private func addData(_ entry: NamedData) throws {
        let archive = self
        let name = entry.name
        let data = entry.data
        do {
            if let data = data {
                try archive.addEntry(with: name, type: .file, uncompressedSize: UInt32(data.count), compressionMethod: .none, provider: { (position, size) -> Data in
                    return data
                })
            }
        } catch {
            throw error
        }
    }

    private func removeData(_ entry: NamedData) throws {
        let archive = self
        let name = entry.name
        do {
            if let entry = archive[name] { try archive.remove(entry) }
        } catch {
            throw error
        }
    }
}

// MARK: - Public
extension Archive {

    /// Update NamedData entries in the archive
    func updateData(entries: [NamedData]) throws {
        // Walk each entry and overwrite
        do {
            for entry in entries {
                try removeData(entry)
                try addData(entry)
            }
        } catch {
            throw error
        }
    }

    /// Add NamedData entries to the archive (updateData is the preferred
    /// method since no harm comes from removing entries before adding them)
    func addData(entries: [NamedData]) throws {
        // Walk each entry and create
        do {
            for entry in entries {
                try addData(entry)
            }
        } catch {
            throw error
        }
    }

    /// Read Data out of the entry using its name
    func readData(named name: String) throws -> Data? {
        let archive = self
        // Get data from entry
        do {
            var entryData : Data? = nil
            if let entry = archive[name] {
                // _ = returned checksum
                let _ = try archive.extract(entry, consumer: { (data) in
                    entryData = data
                })
            }
            return entryData
        } catch {
            throw error
        }
    }
}

Solution

The closure based APIs in ZIPFoundation are designed to provide/consume chunk-wise data. Depending on the final size of your data and the configured chunk size (optional parameter, default is 16*1024), the provider/consumer closures can get called multiple times.

When you are extracting an entry via

let _ = try archive.extract(entry, consumer: { (data) in
    entryData = data
})

you are always overwriting entryData with the latest chunk vended by the consumer closure (if the final size is bigger than the chunk size).

Instead you can use

var entryData = Data()
let _ = try archive.extract(entry, consumer: { (data) in
    entryData.append(data)
})

to make sure that the whole entry gets accumulated in the entryData object.

A similar thing is happening in your Provider code. Instead of always returning the whole image data object, you should provide a chunk (starting at position with size) each time the closure is called.