Search code examples
javadata-cleaningknimebigdata

Finding missing values in big data

I have a number of tables that needs to be analysed. Currently the data is being cleansed as it has a lot of missing values. In some cases there are just one value missing and in other about 200 rows.

Question

How do I find what value should be inserted to the rows?

My Approach

  1. A simple linear extrapolation of the graph.

  2. Finding the mean of the values and inserting it for all missing values.

  3. In case 200 values are missing taking the average of last 100 and next 100 values.

Though I could think of all the above approaches, I am not sure about its efficiency or the method how to write the node to achieve this.

Solution

  • the new KNIME missing value node provides many ways for replacing missing values. It can simply replace by mean or linearly interpolate between the previous and next non-missing values.