Here is my code for CSV data extraction and transformation:
Schema schema = new Schema.Builder()
.addColumnsString("RowNumber")
.addColumnInteger("CustomerId")
.addColumnString("Surname")
.addColumnInteger("CreditScore")
.addColumnCategorical("Geography",Arrays.asList("France","Spain","Germany"))
.addColumnCategorical("Gender",Arrays.asList("Male","Female"))
.addColumnsInteger("Age","Tenure","Balance","NumOfProducts","HasCrCard","IsActiveMember","EstimatedSalary","Exited").build();
TransformProcess transformProcess = new TransformProcess.Builder(schema)
.removeColumns("RowNumber","Surname","CustomerId")
.categoricalToInteger("Gender")
.categoricalToOneHot("Geography").build();
RecordReader reader = new CSVRecordReader(1,',');
reader.initialize(new FileSplit(new ClassPathResource("Churn_Modelling.csv").getFile()));
TransformProcessRecordReader transformProcessRecordReader = new TransformProcessRecordReader(reader,transformProcess);
System.out.println("args = " + transformProcessRecordReader.next() + "");
I just tried printing the first record:
args = [619, 1, 0, 0, 1, 42, 2, 0, 1, 1, 1, 101348.88, 1]
For example, the three values followed by 619 -> 1, 0, 0 I would like to keep 619 followed by 0, 0.
Basically I would like to keep the first category as base category and others are predicted from the base category to avoid any multi-collinear relationship (dummy variable trap)
How do I do that? Can anyone advice on this?
You could check the final transformation schema with transformProcess.finalSchema
, and remove the corresponding 2nd column with
TransformProcess transformProcess = ... same as before...
.categoricalToOneHot("Geography")
.removeColumns("Geography[France]")
.build()