Is it possible to calculate cut and fill volumes in Geoserver from DEM sources? Read value of each pixel and multiply it by pixel size in area would take too much requests.
No, that is not possible in GeoServer. There would be nothing to prevent you writing a WPS function to make it possible though.