How to avoid data leakage and thread blocking while writing data on a file on android

Question

I'm working with android sensors and have a method inside a listener that keeps appending data on a string builder with really high frequency. After some data is collected I compress the string with gzip and write it on a file to avoid out of memory exceptions. This keeps repeating forever. This is all in the same thread so as the file gets bigger it starts to block the thread and the data appending on the string. I do create new files if they get too large but i think i need to implement a threading and lock mechanism for the compression and file writing to avoid any blocking but at the same time not have any problems with leakage of data. Can anyone help me with that? Im not sure if im wording my question correctly.

// on rotation method of gyroscope
                @Override
                public void onRotation(long timestamp,float rx, float ry, float rz) {

                        try  {

                            //get string of new lines of the write data for the sensor
                            str.append("gyroTest,userTag=testUser,deviceTag="+deviceName+" rx="+rx+",ry="+ry+",rz="+rz+" "+timestamp+"\n");

                            if(count >=2000){
                                b = GZIPCompression.compress(str);
                                Log.i(FILE_TAG, "Write gyroscope file");
                                FileHandling.testWrite( GYROSCOPE,b);
                                str.setLength(0);
                                count=0;

                            }
                            count++;


                        } catch (Exception e) {
                            e.printStackTrace();
                        }
                        

                }

Answer 1

You're on the right track in that you need to separate reading from the sensor, processing the data, and writing it all back to disk.

To pass the data from the sensor reads, you may consider using something like a LinkedBlockingQueue with your Strings.

private LinkedBlockingQueue<String> queue = new LinkedBlockingQueue<String>();

@Override
public void onRotation(long timestamp, float rx, float ry, float rz) {
    queue.add(
        "gyroTest,userTag=testUser,deviceTag="+deviceName+" rx="+rx+",ry="+ry+",rz="+rz+" "+timestamp+"\n"
    );
}

And then in another Thread, looping until canceled, you could drain the queue, process, and write without blocking the reading (main) Thread.

private boolean canceled = false;

private void startProcessingQueue() {
    Runnable processQueueRunnable = new Runnable() {
        @Override
        public void run() {
            while (!canceled) {
                drainQueueAndWriteLog();

                Thread.sleep(250);
            }
        }
    };

    new Thread(processQueueRunnable)
        .start();
}

private void drainQueueAndWriteLog() {
    List<String> dequeuedRotations = new ArrayList<String>();

    queue.drainTo(dequeuedRotations);

    if (0 < dequeuedRotations.size()) {
        // Write each line, or all lines together
    }
}

Note: take care to ensure the runnable is canceled when your Activity is paused.

As mentioned in your question, the more data you're writing, the slower it's going to be. Since you're writing data from a sensor, it's inevitably going to grow. For this, you could partition your files into smaller segments, by using something like a date-based naming convention for your log files.

For instance, a log name pattern of yyyyMMddHHmm would create minute-spaced log files, which you could then later aggregate and sort.

private SimpleDateFormat logFileDateFormat = new SimpleDateFormat("yyyyMMddHHmm");

private String getCurrentLogFileName() {
    return String.format(
        "rotations-%s.log",
        logFileDateFormat.format(new Date())
    );
}

Just keep in mind that since you're not writing in the same thread you're reading from, your timestamps may not match up perfectly with your log file names. This shouldn't be a problem, though, as you're already including the timestamps in the persisted data.

Further down the line, if you're still finding you're not quite hitting the level of write-throughput that your project requires, you may also want to consider condensing the amount of information you're actually storing by encoding common byte usages, or even reducing the length of each key to their most-unique values. For example, consider this 1 line output:

"gyroTest,userTag=testUser,deviceTag=some-device-name rx=12345,ry=4567,rz=87901872166251542545144\n"

And now reducing the keys:

"gyroTest,u=testUser,d=some-device-name x=12345,y=4567,z=87901872166251542545144\n"

Removes 18 characters from every line that needs to be written, without sacrificing any information.

Also worth noting: you either need a space (or better a comma) before the timestamp in your data line, else you won't be able to nicely pick out rz from it. And your deviceName should be escaped with quotation marks if it can contain spaces, else it will conflict with pulling out rx.