Reading from a Java InputStream generated by a shell script happens in batches

Question

I am trying to orchestrate Python scripts on a Raspberry Pi through a Java program. I have some Python code with an infinite loop inside that reads the distance from an ultrasonic sensor and outputs it to the console. The Java code calls this as a shell process and then reads the OutputStream the process. The problem I am experiencing is that the data arrives in large batches. Instead of me getting a new value every time the Python script outputs to the console, I get new data every 10 seconds or so. This is problematic because I am trying to display the data on a web dashboard in near-realtime.

The diagram would look like the following:

DistanceController
    |--getDistance(public)
    |--getBufferSize(public)
    |--buffer(private)
    |--DistanceThread(private)
            |--bufferReference (private)
            |--PythonScriptProcess

The Java code is the following:

This is a wrapper class that instantiates a new thread (separate class) to read from the output stream of the Python script.

public class DistanceController {
    private DistanceThread distanceThread;
    private Thread t;
    private LinkedList<DistanceVT>  buffer = new LinkedList<DistanceVT>();

    public DistanceController()
    {            
        t = new Thread(distanceThread);
        t.start();
    }

    public DistanceVT getDistance() {
        return buffer.getLast();
    }
}

This is the thread that reads the distance, the idea is for it to be non-blocking.

public class DistanceThread implements Runnable {
private LinkedList<DistanceVT> buffer;
private String[] args = new String[]{"python", "./run-scripts/distance.py"};

public DistanceThread(LinkedList<DistanceVT> list) {buffer = list;}

@Override
public void run() {
    while (true) {
        try {
            ProcessBuilder pb = new ProcessBuilder(args);
            pb.redirectErrorStream(true);
            final Process p = pb.start();
            while (true) {
                InputStream s = p.getInputStream();
                final BufferedReader reader = new BufferedReader(new InputStreamReader(s));
                while (((line = reader.readLine()) != null)) {
                    buffer.add(parseDistance(line));
                }
            }
        } catch (Exception e) {
        //handle errors
        }
    }
}

And the Python code looks like so (a slightly modified version of the simplest distance measurement code):

import time
import datetime
from random import randint

def measure_distance():
    time.sleep(0.1)
    return randint(10,15)

while True:
    distance = measure_distance()
    print ("%s,%s" % (datetime.datetime.now(),distance))

Answer 1

Thanks you all for your help. EJP solved the buffering problem- adding a sys.stdout.flush() to the Python code resolved the problem.

As to the other questions- you are right. What I meant is that since I am reading on a separate thread, the main thread of the program can carry on doing other logic.

The reading thread accumulates the data in a list that is passed to it by the class that created it. This data can then be accessed through this wrapper class without waiting for the output to complete.

import time
import datetime
from random import randint
import sys

def measure_distance():
    time.sleep(0.1)
    return randint(10,15)

while True:
    distance = measure_distance()
    print ("%s,%s" % (datetime.datetime.now(),distance))
    sys.stdout.flush()