Search code examples
deep-learningdeeplearning4jdl4j

How do I remove dummy variable trap with OneHotencoding


Here is my code for CSV data extraction and transformation:

Schema schema = new Schema.Builder()
            .addColumnsString("RowNumber")
            .addColumnInteger("CustomerId")
            .addColumnString("Surname")
            .addColumnInteger("CreditScore")
            .addColumnCategorical("Geography",Arrays.asList("France","Spain","Germany"))
            .addColumnCategorical("Gender",Arrays.asList("Male","Female"))
            .addColumnsInteger("Age","Tenure","Balance","NumOfProducts","HasCrCard","IsActiveMember","EstimatedSalary","Exited").build();
    TransformProcess transformProcess = new TransformProcess.Builder(schema)
                                            .removeColumns("RowNumber","Surname","CustomerId")
                                            .categoricalToInteger("Gender")
                                            .categoricalToOneHot("Geography").build();
    RecordReader reader = new CSVRecordReader(1,',');
    reader.initialize(new FileSplit(new ClassPathResource("Churn_Modelling.csv").getFile()));
    TransformProcessRecordReader transformProcessRecordReader = new TransformProcessRecordReader(reader,transformProcess);
    System.out.println("args = " + transformProcessRecordReader.next() + "");

I just tried printing the first record:

args = [619, 1, 0, 0, 1, 42, 2, 0, 1, 1, 1, 101348.88, 1]

For example, the three values followed by 619 -> 1, 0, 0 I would like to keep 619 followed by 0, 0.

Basically I would like to keep the first category as base category and others are predicted from the base category to avoid any multi-collinear relationship (dummy variable trap)

How do I do that? Can anyone advice on this?


Solution

  • You could check the final transformation schema with transformProcess.finalSchema, and remove the corresponding 2nd column with

    TransformProcess transformProcess = ... same as before...
            .categoricalToOneHot("Geography")
            .removeColumns("Geography[France]")
            .build()