Search code examples
javascriptd3.jscrossfilterfile-exists

Check for potential CSV file issue in D3


I have a D3 project that uses a very large CSV file to read the data. It works fine except sometimes user sees the spinning loading image indefinitely because of one the following issues.
I need to be able to catch these issues and display some meaningful message to user rather than the never-ending wait message:

  1. When the data file (MyData.csv) doesn’t exist
  2. When the MyData.csv exists but is empty
  3. When the MyData.csv exists, is not empty but is improperly formatted (change the name of a column header on first row for testing).

Ideally, I would like to be able to distinguish among them.

function loadData() {
    d3.csv('MyMap.csv', function(data) {
        data.forEach(function(d) {
            doSomething()
        });

        // load primary data set
        d3.csv('MyData.csv', function (data) {
            doSomethingWithData(data);
        }); // main data set
    });
};

Updated with error checking

var ErrorEnum = Object.freeze({NOTFOUND: 1, EMPTY: 2, INVALIDFORMAT: 3});
var mapFileName = 'MyMap_empty.csv';
var dataFileName = 'MyData_empty.csv';

function redirectOnFileError(fileName, url, errID) {
    var errTitle = errID == ErrorEnum.NOTFOUND ? "File Not Found" : (errID == ErrorEnum.EMPTY ? "Empty File" : "Invalid File Format");
    var errText = fileName + (errID == ErrorEnum.NOTFOUND ? " could not be found" : (errID == ErrorEnum.EMPTY ? " is empty" : " has invalid format"));
    console.log(errText);

    swal({
        title: errTitle,
        text: errText,
        type: "error",
        confirmButtonText: "OK"
        },
        function(isConfirm){
            // somehow we never get here!
            console.log("here ...");
            if (isConfirm) {
                    window.location.href = url;
            }
        }); 
        // hack ... callback function above is not executed, so using below jQuery to use "OK" button's class and 'click' it
        $('.swal2-confirm').click(function(){
            window.location.href = url;
    });
}

function loadData() {
    d3.csv(mapFileName, function (error, data) {
        if (error && error.status === 404) {
            redirectOnFileError(mapFileName, "fnf.html", ErrorEnum.NOTFOUND);
        }
        else if (data.length === 0) {
            redirectOnFileError(mapFileName, "ef.html", ErrorEnum.EMPTY);
        }
        else {
            data.forEach(function (d) {
                // DoSomethingHere();
            });

        // load primary data set
        d3.csv(dataFileName, function (error, data) {
            if (error && error.status === 404) {
                redirectOnFileError(dataFileName, "fnf.html", ErrorEnum.NOTFOUND);
            }
            else if (data.length === 0) {
                redirectOnFileError(dataFileName, "ef.html", ErrorEnum.EMPTY);              }
            else {
                // DomSomethingWithData();
            }); // main data set
    });
}
}
};

Solution

  • Both the first and the second scenarios can be answered here (for the third scenario, see the PS below).

    Before we start, let's see a working, existing CSV file:

     d3.csv("https://gist.githubusercontent.com/GerardoFurtado/2e9f269f7a0f56cf7e23b480afcf280a/raw/5e361753a695534915e81786013976aa94685695/csvfile.csv", function(error, data) {
      console.log(data)
    });
    <script src="https://d3js.org/d3.v4.min.js"></script>

    As you can see, everything works. This is the CSV I'll use in the next snippets.

    First scenario

    Your first scenario...

    When the data file (MyData.csv) doesn’t exist

    ... can be handled using the first parameter in the d3.csv function, which is the error (if any):

    d3.csv(url, function(error,data){...
    //error here ----------^
    

    In the following snippet we're checking for target.status in the error. It will give you a 404, which is printed in the console. I'm just changing the url of the previous CSV file:

    d3.csv("https://gist.githubusercontent.com/GerardoFurtado/2e9f269f7a0f56cf7e23b480afcf280a/raw/fakeurl.csv", function(error, data) {
      if (error && error.target.status === 404) {
          console.log("File not found")
        }
    });
    <script src="https://d3js.org/d3.v4.min.js"></script>

    Second scenario

    Your second scenario...

    When the MyData.csv exists but is empty

    ... can be fixed checking for the length of the data. Here is a demo, using a modified version of my first CSV (without the rows):

    d3.csv("https://gist.githubusercontent.com/GerardoFurtado/2e9f269f7a0f56cf7e23b480afcf280a/raw/5e361753a695534915e81786013976aa94685695/csvfileempty.csv", function(error, data) {
      if (error && error.target.status === 404) {
          console.log("File not found")
        }
      if(data.length === 0){
      console.log("File empty")
      }
    });
    <script src="https://d3js.org/d3.v4.min.js"></script>

    PS: Your third situation ("file is improperly formatted") can be solved only by you: you are the only one who knows the structure of the file.