Thanks in advance for the help.
I am trying to train a binary classifier on a dataset containing about 10000 samples. This dataset is imbalanced with a ratio between the two classes of about 1:9. I would like to increase the cost of misclassification of the lower occurring class but no matter what method I use, nor what cost matrix I use, I get the same results (basically the output of a zeroR model). I would expect to see at least some difference for different cost matrices when using logistic regression which has not been the case. I am using weka 3-7-11. I have noticed that only libLinear (not a stock classifier) gives predictions that are significantly different from zeroR. I have not noticed this with libSVM. This seems very weird to me. What might be going on?
I was able to figure out what was going on. @FilipMalczak was partially correct. You cannot use the cost matrix tool under the "More Options" button; it doesn't work. Instead, you have to pick meta/CostSensitiveClassify as your classifier and then within the options for this (reached by clicking on the name of the classifier; not within the pulldown menu mind you) assign the cost matrix and actual classifier you want.