Search code examples
phpjsondecoding

Maximum json file size which can be decoded without failure


I have a php array structure

Array
(
    [car] => Array
        (
            [red] => 0.333
            [orange] => 0.333
            [blue] => 0.333
        )

    [truck] => Array
        (
            [white] => 0.333
            [green] => 0.333
            [blue] => 0.333
        )
)

I have been using serialize to save the array to a text file, and unserialize to get back the array form. Unfortunately, the serialized array is getting very large, but mostly due to the float point (bug or by design) conversion when serialized. For example, instead of 0.333, the serialized process converts .333 into .3333333333333333333333333333333333333333333333333. This made me want to switch to json_encode to save the array. When compare serialize to json_encode, the serialized file is 40MB in size vs 8MB size for json_encode.

That is great, except when I try to json_decode the file, it is no longer in the array form. I tried json_decode($array, true), but that does not work either.

How can I get json_encode to work for this example?

PS, my floating point number was generated by rounding off the decimals. Another answer that I found on Stack Overflow suggested that instead of using round($part/$sum, 3);, use sprintf('%.3f', $part/$sum); which turned the floating point into a string. That alone cut the serialized file down from 40MB to 19MB, but it still much larger than the json_encode file size of 8MB.


Solution

  • The 'problem' is due to json_decode inability to read large json_encode files. The largest json file that can work is only ~.5MB. Tested on 4GB Ram, 4 core Xeon server, and also 4gb localhost laptop. I also had set memory_limit in the php.ini file to be 3GB for other php routines (yes, 3GB) and restarted apache. So the memory_limit setting appears not to be the problem.

    Error message was not helpful, it stated that

    Warning: array_slice() expects parameter 1 to be array, null given in /home/xxxxx/public_html/xxxx.php on line xx

    Hopefully this error message would help some person in the future to narrow down the bug.