OVH Object Storage, nothing happens when I try to upload large file (more than 100 Ko)

Question

I try to upload files with OVH object storage. But I have three different behavior according to the heavy file.

• With a weight file which less than 100Ko, everything is ok

• With a weight file which more than 100Ko, I have this error: Error: write after end, but the file is uploaded on ovh object storage

• With a weight file which more than 250Ko, nothing happens, and the file is not uploaded. The fs ReadStream is open, but the write stream piped (with the read stream) not finish.

This is my code:

  var client = require('pkgcloud').storage.createClient({
    provider: 'openstack',
    username: myusername,
    password: mypassword,
    region: 'GRA',
    authUrl: 'https://auth.cloud.ovh.net/'
  });

  const fsReadStream = fs.createReadStream(path.resolve(__dirname, fileLocation))

  let writeStream = client.upload({
    container: myOvhStorageContainer,
    remote: 'fileName.jpg',
  });

  writeStream.on('error', function (err) {

    console.log(err)
  });

  writeStream.on('success', async function (file) {
    console.log(file)
  });

  fsReadStream.on('open', function () {
    console.log('open!!')

    fsReadStream.pipe(writeStream);
  });

Answer 1

The problem comes from a bug in pkgcloud in how it streams files for OpenStack Storage.

The solution is exposed, and a fix is proposed, in https://github.com/pkgcloud/pkgcloud/pull/673

There are forks of pkgcloud that include the proposed fix, and they can be used while waiting for the fix to be officially accepted:

• https://github.com/madarche/pkgcloud/
• https://github.com/StartupFlow/pkgcloud/tree/fix_write_after_end

To use such a repository in your project/application, edit your package.json to modify the dependencies like this:

"dependencies": {
    …
    "pkgcloud": "https://github.com/madarche/pkgcloud.git#fe2701eb6eb984e3d067d5df7610ca0751528dbd",
    …
  },

You can also simply create your own fork of pkgclould, so you don't have to trust a random Git repository.