Firebase Cloud FireStore: Insert Large Array

Question

Disclaimer: I have been programming for about 4 months now, so I'm still very new to programming.

I am using Firebase Cloud Firestore for a database and inserting CSV files with large amounts of data in them. Each file can be about 100k records in length. The project I'm working on requires a user to upload these CSV's from a web page.

I created a file uploader and I'm using the PapaParse JS tool to parse the csv, and it does so very nicely, it returns an array instantly, even if it's very long. I've tried to use the largest files I can find and then logging it to the console, it's very fast.

The problem is when I then take that array it gives me and loop through it and insert that into Cloud Firestore. It works, the data is inserted exactly how I want it. But it's very slow. And if I close the browser window it stops inserting. Inserting only 50 records takes about 10-15 seconds. So with files of 100k records, this is not going to work.

I was considering using Cloud Functions, but before I now try and learn how all that works, maybe I'm just not doing this in an efficient way? So I thought to ask here.

Here is the JS

// Get the file uploader in the dom
var uploader = document.getElementById('vc-file-upload');

// Listen for when file is uploaded 
uploader.addEventListener('change',function(e){
		
		// Get the file 
		var file = e.target.files[0];

		// Parse the CSV File and insert into Firestore
		const csv = Papa.parse(file, {
			header: true,
			complete: function(results) {
				console.log(results);
				
				sim = results.data;
				var simLength = sim.length;
				for (var i = 0; i < simLength;i++) {
					var indSim = sim[i]
					iccid = indSim.iccid;
					const docRef = firestore.collection('vc_uploads').doc(iccid);
					docRef.set({indSim}).then(function() {
					console.log('Insert Complete.')
					}).catch(function (err) {
					console.log('Got an error: '+ err)
				 	})
				};

			}
		});
});

Answer 1

It will almost certainly be faster overall if you just upload the file (perhaps to Cloud Storage) and perform the database operations in Cloud Functions or some other backend, and it will complete even if the user leaves the app.