Cannot reliably write to avro from resultset of teradata

Question

My code in java is reading teradata using result set and is printing every row in the terminal but it isn't writing correctly to avro i.e. few of the rows are dropping for no reason. There is no pattern in which dropping of row is happening and I'm converting every data type into String while inserting into avro file. I feel there is an error in Avro writing process on my end but can't figure it out. Any help is appreciated. Here is the code.

`

static avroWriter(Schema schema, OutputStream outStream, ResultSet rs) {
    final GenericRecord rec = new GenericData.Record(schema)
    final DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<>(schema)
    final DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<>(datumWriter)
    dataFileWriter.create(schema, outStream)
    final int nrOfColumns = rs.getMetaData().getColumnCount()
    while (rs.next()) {
        for (int i = 1; i <= nrOfColumns; i++) {
            final Object colValue = rs.getObject(i)
            String value
            if(colValue == null){
                value = "null"
            }else{
                value = colValue.toString()
            }
            //println("i  :" + i + " value : "+ value)
            rec.put(i - 1,value)
        }
        try {
            dataFileWriter.append(rec)
            println(rec)
        }
        catch (IOException e) {
            log.error("Record :{} couldn't be read properly", rec, e)
        }
    }
}

`

Answer 1

End up solving it. Flushing the data everytime you write a record seems to works fine for me.

dataFileWriter.append(rec) dataFileWriter.flush()