As a university project, we have been tasked to perform a 14-hour long experiment, where we need to record changing data from the LDR photoresistor. We are using A Huzzah32 esp32 with the 4 megabytes of disk space and programming it using Micro Python. We need to leave the machines conducting experiments and gather data from them.
(!) For every measurement, the sensor yields an integer ranging from 0 to 4095.
I would like to know, what is the most efficient way to store the experiment data on the machine? The measurements can be repeated very frequently. Is it a good idea to write a compression algorithm and store them as compressed files? Or is it no efficient to put any additional .py scripts? The specs of my machine are linked.
If the rate is 10 Hz, as indicated in the comments, then you don't need to compress. Storing two bytes per sample will take about a megabyte, and you have two (based on @Lixas comment). So you could go over 20 Hz with no compression.
For higher rates, the first improvement would be to store two samples in every three bytes, since each sample is 12 bits. That will take you to 27 Hz.
To go further, there would need to be some correlation between values. For example, if the light levels change slowly, the differences between successive measurements will be small compared to the full range of 0..4095. You can then take the difference of every sample from the last, and store that instead. You should take a lot of measurements in a representative experiment scenario and make a histogram of the differences. From that you can decide how many bits to allocate a difference that will cover it most of the time. Then you can have a scheme to encode short and long differences. For example, if the differences are almost always less than ±32, then you could get another factor of two compression, and get to over 50 Hz.