get data asynchron from Rest api and loading them to mongodb with node.js - using promises

Question

I worked with Javascript not that much and struggle now.

In Short:

I want to retrieve data from a Rest-API and bulk insert them into a collection of a running MongoDB database.

My knowledge so far:

As Javascript is a single-threaded language. So if you want that something is done after other things are finished (mostly async operations), you have to use callbacks. Callback: A Callback is a function A which is passed to a function as a param and must be executed at the end inside that function to guarantee that either A

• is executed as the latest action

or

• data/objects which had to be retrieved async are possibly available so they could be used inside A

Promises:

Is a relatively new feature introduced to Javascript which could be used to deal with Javascript single-threaded nature? Promises have different states (fulfilled, rejected, pending, settled).

async/await: Are also a new feature to let working with asynchronous functions look more synchronous and to prevent too much nesting which could lead to confusion.

My scenario:

This is a class with a function which uses 'request' to do a REST-Request.

class DataManager {
    static loadSkill(uri) {
        return new Promise((resolve, reject) => {
            const apiEndpoint = 'apiEndpoint';
            request(apiEndpoint + uri, (error, response, body) => {
                if (!error && response.statusCode === 200) {
                    resolve(body)
                } else reject(error);
            });
        });
    }
 ...
}

In the following function, I read the URIs of the data I want to retrieve from a JSON file. The loaded and parsed array is split in arrays of the size of 100 entries because I want to bulk insert 100 objects at once later. In the inner forEach loop, I try to load the skill by using the promise with await, so the function has to be declared with async

function retrieveAndInsertAllSkills() {
    //DBManager.createCollection('skills');
    fs.readFile('./data/uris.json', 'utf8', function readFileCallback(err, data) {
        if (err) {
            console.log(err);
        } else {
            let obj = JSON.parse(data);

            let chunks = chunk(obj.uris, 100);

            chunks.forEach((chunk)=>{
                let skills = [];
                chunk.forEach(async (uri) => {
                    let skill = await Datamanager.loadSkill(uri);
                    skills.push(skill);
                });
                DBManager.insertMany(skills,'skills')
            })
        }
    });
}

At first, I run in an error regarding MongoDB. But I realized that the error was caused by trying to insert empty arrays. So, to be honest, I ran in a similar problem at another problem.

I try to execute a Linux shell command (sha1sum [file]) to get a checksum. The files are large and I

class FileUtil {
    static executeCommand(command, paramsArray) {
        return new Promise((resolve, reject) => {
            let cmd = spawn(command, paramsArray);
            cmd.stdout.on('data', data => {
                resolve(data.toString());
            });
            cmd.stderr.on('data', error => {
                reject(error.toString());
            });
        });
    }
  ...
    static getSha1Sum(file){
        this.executeCommand('sha1sum', [file])
            .then((chkSum) => {
                console.log(chkSum);
                return chSum
            })
            .catch((errorMsg) =>{
                console.log(errorMsg);
                return new Error(errorMsg)});
    };
 ...
}

The following code prints 'undefined' at the console only. The code above prints the desired result.

async function testcheck(file) {
    try {
        let hash = await FileUtil.getSha1Sum(file);
        console.log(hash);
    }
    catch (e) {
        console.log(e);
    }
}
testcheck('/home/coloeus/Downloads/WebStom-2018.2.2.tar.gz');

So my question is: What's wrong with my code. I hope you can give me a hint. I have read a lot about using promises but can't figure out why it isn't working.

Answer 1

replace the second forEach with for..of, like so:

chunks.forEach(async (chunk)=>{
    let skills = [];
    for (const uri of chunk) skills.push(await Datamanager.loadSkill(uri));
    DBManager.insertMany(skills,'skills');
})