Search code examples
pythonmachine-learningscikit-learnsvmmulticlass-classification

How to draw the hyperplanes for SVM One-Versus-All?

I was trying to draw the hyperplanes when SVM-OVA was performed as following:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.svm import SVC
x = np.array([[1,1.1],[1,2],[2,1]])
y = np.array([0,100,250])
classifier = OneVsRestClassifier(SVC(kernel='linear'))

Based on the answer to this question Plot hyperplane Linear SVM python, I wrote the following code:

fig, ax = plt.subplots()
# create a mesh to plot in
x_min, x_max = x[:, 0].min() - 1, x[:, 0].max() + 1
y_min, y_max = x[:, 1].min() - 1, x[:, 1].max() + 1
xx2, yy2 = np.meshgrid(np.arange(x_min, x_max, .2),np.arange(y_min, y_max, .2))
Z = classifier.predict(np.c_[xx2.ravel(), yy2.ravel()])
Z = Z.reshape(xx2.shape)
ax.contourf(xx2, yy2, Z, cmap=plt.cm.winter, alpha=0.3)
ax.scatter(x[:, 0], x[:, 1], c=y, cmap=plt.cm.winter, s=25)

# First line: class1 vs (class2 U class3)
w = classifier.coef_[0]
a = -w[0] / w[1]
xx = np.linspace(-5, 5)
yy = a * xx - (classifier.intercept_[0]) / w[1]
ax.plot(xx,yy)

# Second line: class2 vs (class1 U class3)
w = classifier.coef_[1]
a = -w[0] / w[1]
xx = np.linspace(-5, 5)
yy = a * xx - (classifier.intercept_[1]) / w[1]
ax.plot(xx,yy)

# Third line: class 3 vs (class2 U class1)
w = classifier.coef_[2]
a = -w[0] / w[1]
xx = np.linspace(-5, 5)
yy = a * xx - (classifier.intercept_[2]) / w[1]
ax.plot(xx,yy)

However, this is what I obtained:

enter image description here

The lines are clearly wrong: actually, the angular coefficients seem correct, but not the intercepts. In particular, the orange line would be correct if translated by 0.5 down, the green one if translated by 0.5 left and the blue one if translated by 1.5 up.

Am I wrong to draw the lines, or the classifier does not work correctly because of the few training points?

Solution

  • The problem is the C parameter of SVC is too small (by default 1.0). According to this post,

    Conversely, a very small value of C will cause the optimizer to look for a larger-margin separating hyperplane, even if that hyperplane misclassifies more points.

    Therefore, the solution is to use a much larger C, for example 1e5

    import matplotlib.pyplot as plt
import numpy as np
from sklearn.svm import SVC
from sklearn.multiclass import OneVsRestClassifier


x = np.array([[1,1.1],[1,2],[2,1]])
y = np.array([0,100,250])
classifier = OneVsRestClassifier(SVC(C=1e5,kernel='linear'))
classifier.fit(x,y)

fig, ax = plt.subplots()
# create a mesh to plot in
x_min, x_max = x[:, 0].min() - 1, x[:, 0].max() + 1
y_min, y_max = x[:, 1].min() - 1, x[:, 1].max() + 1
xx2, yy2 = np.meshgrid(np.arange(x_min, x_max, .2),np.arange(y_min, y_max, .2))
Z = classifier.predict(np.c_[xx2.ravel(), yy2.ravel()])
Z = Z.reshape(xx2.shape)
ax.contourf(xx2, yy2, Z, cmap=plt.cm.winter, alpha=0.3)
ax.scatter(x[:, 0], x[:, 1], c=y, cmap=plt.cm.winter, s=25)

def reconstruct(w,b):

    k = - w[0] / w[1]
    b = - b[0] / w[1]

    if k >= 0:
        x0 = max((y_min-b)/k,x_min)
        x1 = min((y_max-b)/k,x_max)
    else:
        x0 = max((y_max-b)/k,x_min)
        x1 = min((y_min-b)/k,x_max)
    if np.abs(x0) == np.inf: x0 = x_min
    if np.abs(x1) == np.inf: x1 = x_max
    
    xx = np.linspace(x0,x1)
    yy = k*xx+b

    return xx,yy

xx,yy = reconstruct(classifier.coef_[0],classifier.intercept_[0])
ax.plot(xx,yy,'r')
xx,yy = reconstruct(classifier.coef_[1],classifier.intercept_[1])
ax.plot(xx,yy,'g')
xx,yy = reconstruct(classifier.coef_[2],classifier.intercept_[2])
ax.plot(xx,yy,'b')

    This time, because a much larger C is adopted, the result looks better

    output