How to write multiple avro objects into a ByteArrayOutputStream

Question

The Avro documentation page has an example that write multiple avro objects into a file:

DatumWriter<User> userDatumWriter = new SpecificDatumWriter<User>(User.class);
DataFileWriter<User> dataFileWriter = new DataFileWriter<User>(userDatumWriter);
dataFileWriter.create(user1.getSchema(), new File("users.avro"));
dataFileWriter.append(user1);
dataFileWriter.append(user2);
dataFileWriter.append(user3);
dataFileWriter.close();

I would like to achieve something similar but write to a ByteArrayOutputStream.

ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
Encoder encoder = EncoderFactory.get().binaryEncoder(outputStream, null);
DatumWriter<MyData> myDataDatumWriter = new SpecificDatumWriter<>(MyData.getClassSchema());

myDataDatumWriter.write(myData1, encoder);
encoder.flush();
myDataDatumWriter.write(myData2, encoder);
encoder.flush();

I sent the data in the output stream as a http response, but the data cannot be deserialized on the client side because it is not a recognizable avro format.

What's wrong with this code?

Answer 1

You just need to create a SeekableByteArrayInput from your byte array, and then read it using the DataFileReader, e.g.:

public static void main(String[] args) throws IOException {
    final ByteArrayOutputStream baos = new ByteArrayOutputStream();

    // First write the records to the ByteArrayOutputStream.

    final DatumWriter<User> dataWriter = new SpecificDatumWriter<>(User.class);
    try (final DataFileWriter<User> dataFileWriter =
                 new DataFileWriter<>(dataWriter).create(User.SCHEMA$, baos)
    ) {
        dataFileWriter.append(myData1);
        dataFileWriter.append(myData2);
        dataFileWriter.append(myData3);
    }

    // Now read the records back in from the byte array.

    final DatumReader<User> dataReader = new SpecificDatumReader<>(User.class);

    final List<User> recs = new ArrayList<>();
    try (
            final SeekableInput seekIn = new SeekableByteArrayInput(baos.toByteArray());
            final DataFileReader<User> dataFileReader = new DataFileReader<>(seekIn, dataReader)
    ) {
        while (dataFileReader.hasNext()) {
            recs.add(dataFileReader.next());
        }
    }

    assert(recs.size() == 3);
}