Serving large rasters using GeoServer results in JAVA HEAP out of memory

I have rasters that are extremely large like 35000px X 30000px on GeoServer.(Mosaic images) What i'm currently doing is downloading these images using WMS and after that executing analysis on this image, modifying it(I am coloring some segments of that image) and afterwards im uploading it back to GeoServer as new layer.

This works great for images up to 20000px X 20000px, but after that GeoServer cant merge them and instead of geotiff image returns file with xml content and inside it:

ERROR [geoserver.ows] - java.lang.OutOfMemoryError: Java heap space

Now in java options variable i inserted Xmx to be 22 GB and inside script for running jetty Xmx is placed to be 20GB. I was closely watching average load using top and free -m command and at no time more than 8GB of memory was being used. My server has 28GB RAM and 8 processors, and CPU Load never went more than 700%. Can someone give me a hint at what i am doing wrong and what can i do so i can download such big Geotiff images. Btw i am using jetty as a server.

Solution

Although i didn't realise why does this error occur i managed to download whole raster from GeoServer using Web coverage service (WCS) instead of WMS. WCS is the right way to go when downloading big raster images. So if anyone stumbles upon similar problem i recommend you to send post request to wcs service on your geoserver like this POST http://192.168.4.162:9090/geoserver/wcs and as data pass xml describing your layer(You can create it using WCS request builder under demos menu in Geoserver).Mine looked like this:

<?xml version="1.0" encoding="UTF-8"?><GetCoverage version="1.0.0" service="WCS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wcs" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xsi:schemaLocation="http://www.opengis.net/wcs http://schemas.opengis.net/wcs/1.0.0/getCoverage.xsd">
  <sourceCoverage>fiware:test</sourceCoverage>
  <domainSubset>
    <spatialSubset>
      <gml:Envelope srsName="EPSG:4326">
        <gml:pos>-87.3539257 38.3719729154</gml:pos>
        <gml:pos>-87.3437449312 38.3783359</gml:pos>
      </gml:Envelope>
      <gml:Grid dimension="2">
        <gml:limits>
          <gml:GridEnvelope>
            <gml:low>0 0</gml:low>
            <gml:high>32768 20479</gml:high>
          </gml:GridEnvelope>
        </gml:limits>
        <gml:axisName>x</gml:axisName>
        <gml:axisName>y</gml:axisName>
      </gml:Grid>
    </spatialSubset>
  </domainSubset>
  <output>
    <crs>EPSG:4326</crs>
    <format>GeoTIFF</format>
  </output>
</GetCoverage>

And dont forget to put application/xml as Content-type. Cheers!