Speeding up array inserts in NeDB

Question

I'm making an app that grabs an user's comments from a reddit API and loads them in a local database. I'm using NeDB for the database. The reddit API returns an array of 25 elements(comment) and then you have to paginate using the date of the last element to grab the next array of elements. I map every array and insert each element individually. I don't want to insert the whole array because then the elemens get indexed and I would be left with a database full of different indexes. I'm usually grabbing a few hundred arrays. I'm also using nedb-promises.

A whole array insert goes from an average speed of 3 seconds to 10 seconds

The thing is, the array inserts take too long, and the process gets slower at the end. Is there something wrong with my code? Would using SQLite be much faster? Thanks

const asyncForEach = async function (array, callback) {
    for (let index = 0; index < array.length; index++) {
      await callback(array[index], index, array);
    }
}


const fetch = async (paginationHelper) => {
    //response tree: response.data.data[n].body
    const {data:{data}} = await pushshift.get(`?before=${paginationHelper !==null ? paginationHelper : ''}&author=${author}`)

    return data.map(comment => {
        return ({
            body: comment.body,
            created: comment.created_utc,
            subreddit: comment.subreddit,
            author: comment.author,
            postId: comment.link_id.split('t3_')[1],
            parentId: comment.parent_id.split('t1_')[1],
            linkId: comment.id      
        })
    })
}

const load = async (comments) => {
    let i = 0
    await asyncForEach(comments, async comment => {
        await db.insert(comment).then(res => {
            console.log(res);
            i++
            console.log(i);            
        })        
    })
}

const loadAll = async () => {
    let comments = await fetch(null)

    //Pagination is made using the comments' date when they were created
    let paginationHelper = comments[comments.length-1].created

    while(paginationHelper){
        [comments] = await Promise.all( [fetch(paginationHelper), load(comments)] )

        paginationHelper = comments.length !== 0 ? comments[comments.length-1].created : null
        console.log(paginationHelper);
    }
}

Answer 1

I'm an idiot, you can just insert the whole array without any issues and its crazy fast.