Search code examples
javascriptnode.jsweb-scrapingosmosis

node.js Osmosis write results to file

I have scraped multiple sites with the same structure successfully. When I print the results they seem OK, now I want to save it all to a file.

Problem is only the last scraped MyData-object gets saved to the file.

var osmosis = require('osmosis');
var jsonfile = require('jsonfile')
var sitesToHandle = ['site1', 'site2', 'site3', 'site4']
sitesToHandle.forEach((urlToHandle) => {
osmosis.get(urlToHandle)
.find('.productList')
.set({
  MyData: [
    {
        'ID': 'a.number',
        'Product': 'a.productname',
        'Price': 'a.price',
    } 
  ]
})
.data(function(document) {
  console.log(document);
    var file = 'osmosis.json'
    jsonfile.writeFile(file, document)
  });
});
Solution

  • you are rewriting your file each iteration.

    you have to use {flag: 'a'} for appending data to existing JSON file:

    jsonfile.writeFile(file, document, {flag: 'a'})