Is there a way to find out the size of the image (in bytes, in whatever compressed image format they're stored in) in PDImageXObject without extracting it into BufferedImage?
Call img.getCOSObject().getLength(), this will give you the length of the COSStream on which the image is based. If the image has a mask, you'll have to do the same with it. Call img.getMask() to check whether there is one.