Writing a single file to multiple s3 buckets with gulp-awspublish

Question

I have a simple single-page app, that is deployed to an S3 bucket using gulp-awspublish. We use inquirer.js (via gulp-prompt) to ask the developer which bucket to deploy to.

Sometimes the app may be deployed to several S3 buckets. Currently, we only allow one bucket to be selected, so the developer has to gulp deploy for each bucket in turn. This is dull and prone to error.

I'd like to be able to select multiple buckets and deploy the same content to each. It's simple to select multiple buckets with inquirer.js/gulp-prompt, but not simple to generate arbitrary multiple S3 destinations from a single stream.

Our deploy task is based upon generator-webapp's S3 recipe. The recipe suggests gulp-rename to rewrite the path to write to a specific bucket. Currently our task looks like this:

gulp.task('deploy', ['build'], () => {
  // get AWS creds
  if (typeof(config.awsCreds) !== 'object') {
    return console.error('No config.awsCreds settings found. See README');
  }

  var dirname;
  const publisher = $.awspublish.create({
    key: config.awsCreds.key,
    secret: config.awsCreds.secret,
    bucket: config.awsCreds.bucket
  });

  return gulp.src('dist/**/*.*')
    .pipe($.prompt.prompt({
      type: 'list',
      name: 'dirname',
      message: 'Using the ‘' + config.awsCreds.bucket + '’ bucket. Which hostname would you like to deploy to?',
      choices: config.awsCreds.dirnames,
      default: config.awsCreds.dirnames.indexOf(config.awsCreds.dirname)
    }, function (res) {
      dirname = res.dirname;
    }))
    .pipe($.rename(function(path) {
        path.dirname = dirname + '/dist/' + path.dirname;
    }))
    .pipe(publisher.publish())
    .pipe(publisher.cache())
    .pipe($.awspublish.reporter());
});

It's hopefully obvious, but config.awsCreds might look something like:

awsCreds: {
  dirname: 'default-bucket',
  dirnames: ['default-bucket', 'other-bucket', 'another-bucket']
}

Gulp-rename rewrites the destination path to use the correct bucket.

We can select multiple buckets by using "checkbox" instead of "list" for the gulp-prompt options, but I'm not sure how to then deliver it to multiple buckets.

In a nutshell, if $.prompt returns an array of strings instead of a string, how can I write the source to multiple destinations (buckets) instead of a single bucket?

Please keep in mind that gulp.dest() is not used -- only gulp.awspublish() -- and we don't know how many buckets might be selected.

Answer 1

Never used S3, but if I understand your question correctly a file js/foo.js should be renamed to default-bucket/dist/js/foo.js and other-bucket/dist/js/foo.js when the checkboxes default-bucket and other-bucket are selected?

Then this should do the trick:

// additionally required modules
var path = require('path');
var through = require('through2').obj;

gulp.task('deploy', ['build'], () => {
  if (typeof(config.awsCreds) !== 'object') {
    return console.error('No config.awsCreds settings found. See README');
  }

  var dirnames = []; // array for selected buckets
  const publisher = $.awspublish.create({
    key: config.awsCreds.key,
    secret: config.awsCreds.secret,
    bucket: config.awsCreds.bucket
  });

  return gulp.src('dist/**/*.*')
    .pipe($.prompt.prompt({
      type: 'checkbox', // use checkbox instead of list
      name: 'dirnames', // use different result name
      message: 'Using the ‘' + config.awsCreds.bucket + 
               '’ bucket. Which hostname would you like to deploy to?',
      choices: config.awsCreds.dirnames,
      default: config.awsCreds.dirnames.indexOf(config.awsCreds.dirname)
    }, function (res) {
      dirnames = res.dirnames; // store array of selected buckets
    }))
    // use through2 instead of gulp-rename
    .pipe(through(function(file, enc, done) {
      dirnames.forEach((dirname) => {
        var f = file.clone();
        f.path = path.join(f.base, dirname, 'dist',
                           path.relative(f.base, f.path));
        this.push(f);
      });
      done();
    }))
    .pipe(publisher.cache())
    .pipe($.awspublish.reporter());
});

Notice the comments where I made changes from the code you posted.

What this does is use through2 to clone each file passing through the stream. Each file is cloned as many times as there were bucket checkboxes selected and each clone is renamed to end up in a different bucket.