Optimizing Stream-to-JSON Conversion for Performance and Memory Efficiency

Question

I have a Ballerina program where I'm converting a stream of records to JSON using a custom convertToJson function. The current implementation involves looping through each record and invoking the toJson method, resulting in a noticeable performance bottleneck and high memory consumption.

import ballerina/io;

public function main() returns error? {
    stream<ScanRecord> scanRecords = getScanRecordStream(); // large stream
    json[] jsonData = convertToJson(scanRecords);
    io:println(jsonData);
}

type ScanRecord record {
    int id;
    string name;
};

function convertToJson(stream<ScanRecord> scanRecords) returns json[] {
    json[] output = [];
    scanRecords.forEach(function(ScanRecord scanRecord) {
        output.push(scanRecord.toJson());
    });
    return output;
}

The current approach takes approximately 4 seconds for execution, and I'm looking for a more efficient way to directly convert the stream to JSON without iterating through each record. Are there any Ballerina features or optimizations that could help improve the performance and memory efficiency of this operation?

Answer 1

If the ScanRecord only contains fields that are compatible with JSON, the conversion process can be optimized by making it a subtype of json.

Here are two approaches:

1. Make it as closed Record: Make ScanRecord a closed record where all fields are explicitly mentioned

type ScanRecord record {|
    int id;
    string name;
|};

2. Make it open as record with JSON rest descriptor: Make ScanRecord an open record, allowing for extra fields, but explicitly specify the rest descriptor as json. This ensures that any additional fields also belong to json.

type ScanRecord record {|
    int id;
    string name;
    json...;
|};

With these approaches, The ScanRecord becomes a subtype of json, enabling direct use where a json value is expected without converting using toJson().

Example

function convertToJson(stream<ScanRecord> scanRecords) returns json[] {
    return from ScanRecord rec in scanRecords select rec;
}