Search code examples
swiftmacosfoundationswift-niosoto

How do I get `Data` objects into Swift-NIO without making copies?

I'm fairly new to Swift and very new to NIO.

I'm adding Swift code to a large project that needs to up/down load a lot of data (GBs) to AWS. To that end, I've imported the GitHub project Soto, which relies heavily on NIO.

Most methods that send/receive data do so through ByteBuffer structs. My application already has the data to upload in Foundation Data objects. I'm having trouble figuring out the best way to get these Data objects into NIO.

In the documentation for NIO's ByteBuffer (2.26.0) it states

Supported types: A variety of types can be read/written from/to a ByteBuffer. ... Out of the box, ByteBuffer supports for example the following types (non-exhaustive list):

  • String/StaticString
  • Swift’s various (unsigned) integer types
  • Foundation‘s Data
  • [UInt8] and generally any Collection of UInt8

However, the latest swift-nil package has no ByteBuffer support for Foundation Data objects. Instead, it supports DispatchData objects, which in turn seem to have no interoperability with Data objects.

What I want to avoid is making a copy of every block of data (100's of MB at a time), just to convert between Data and DispatchData types.

So...

Right now my thinking is one of

  • I'm completely lost, and there's a simple solution I haven't found

  • The solution is to create a subclass of DispatchData backed by a Data object

  • Initialize the ByteBuffer structure using a DispatchData created using the no-copy initializer pointing to the raw byte array in the Data object, along with a custom deallocator that simply retains the Data object until the ByteBuffer and DispatchData objects are destroyed.

I would appreciate any thoughts, experience, or suggestions (particularly if it's option #1).

Solution

  • You'll need to import NIOFoundationCompat to get any of NIO's method that work with Foundation data types such as Data (or JSONDecoder/JSONEncoder). NIOFoundationCompat is just another module of the swift-nio package so you won't need another dependency.

    But just to be clear, under the hood, there will always be copies but probably you don't need to worry about them, copies are extremely fast on today's CPUs. If you absolutely want to avoid copies, you'll need to create ByteBuffers straight away. To help you with that, you may want to add where you get your data from that you want to send over the network.