NodeJs sharp Image library - Resize using Stream Api throws error 'stream.push() after EOF'

Question

It's bit strange to me the way stream is behaving sometime. Simple task at work is to pull a image from S3 and transform it using sharp library; finally upload resized image back to S3. I can see resized image in S3 but code fails with error.

Here is a code snippet doing all the work -

// Read from S3
const readStream = s3Handler.readStream(params)

// resize Stream
let resizeStream = sharp()
resizeStream
    .metadata()
    .then((metadata) => {

        // Resize the image to a width specified by the `percent` value and output as PNG
        sharpOptions = { width: width*(metadata.percent), height: height*(metadata.percent), fit: 'contain' }
        
        resizeStream
            .resize(sharpOptions)
            .toFormat('png')
            .png({ quality: 100, compressionLevel: 5 })
            .toBuffer()

        //return streamResize.resize(sharpOptions).png().toBuffer()
    })

// push to s3
const { writeStream, uploaded } = s3Handler.writeStream({
    Bucket: bucket,
    Key: key,
    Ext: ext,
})


readStream.pipe(resizeStream).pipe(writeStream)

await uploaded

Here is the error I get from above code -

{
    "errorType": "Error",
    "errorMessage": "stream.push() after EOF",
    "code": "ERR_STREAM_PUSH_AFTER_EOF",
    "stack": [
        "Error [ERR_STREAM_PUSH_AFTER_EOF]: stream.push() after EOF",
        "    at readableAddChunk (_stream_readable.js:260:30)",
        "    at Sharp.Readable.push (_stream_readable.js:213:10)",
        "    at Object.<anonymous> (/var/task/node_modules/sharp/lib/output.js:863:18)"
    ]
}

Any suggestions much appreciated.

Answer 1

You're trying to do something which is currently a bit hard with node-sharp: see https://github.com/lovell/sharp/issues/236

In short, the problem is that .metadata() requires reading the entire image into memory (as a Buffer) to compute values. In addition to that, relative resizing (percentage of original size) creates a data dependency - you need to know the image dimensions before you can compute the target width and height.

Given how you're using Sharp as a pass-through stream (S3 read → resize → S3 write), calling .metadata() on it will not work, because it happens too late: you need to set up the resize options before you pipe the data in (so that the resize process may know the target size). At the same time, you need to pipe all the data in first, because .metadata() needs the complete image. The two needs conflict and make it impossible to have a single Sharp pass-through Stream that does everything.

You're left with a few options:

1. Resign from metadata-based dynamic resizing and use a static maximum size (or a size that's known from another source - submitted by client, loaded from a config, etc.)
2. Load the entire image into a Buffer, compute the metadata from that, and then resize the whole buffer
3. Determine the image size using another streaming technique that does not need the whole data at once and run the pipeline when you've got the size, like this

If you're not dealing with very big images, 2. is decidedly the easiest way to implement this:

const original = (await s3.getObject(getParams).promise()).Body;
const metadata = await sharp(original).metadata();
const resized = await sharp(original).resize({
  // NOTE: This computation was bugged in the original example - there's no metadata.percent, it's our own setting.
  width: metadata.width * percent,
  height: metadata.height * percent,
  fit: 'contain'
}).toFormat('png').png({ quality: 100, compressionLevel: 5 }).toBuffer()
await s3.putObject({
  ...putParams,
  Body: resized
});