I have the following dataset:
node bc cluster
1 russian 0.457039 1
48 man 0.286875 1
155 woman 0.129939 0
3 bit 0.092721 0
5 write 0.065424 0
98 age 0.064347 0
97 escap 0.062675 0
74 game 0.062606 0
Then I perform kMeans clustering by bc value to separate the nodes into two different groups. Right now with the code below I get the result above (the clustering result is in the cluster column).
bc_df = pd.DataFrame({"node": bc_nodes, "bc": bc_values})
bc_df = bc_df.sort_values("bc", ascending=False)
km = KMeans(n_clusters=2).fit(bc_df[['bc']])
bc_df.loc[:,'cluster'] = km.labels_
print(bc_df.head(8))
Which is pretty good, but I would like it to work slightly differently and to select the first 4 nodes into the first cluster and then the other ones in the 2nd one, because they are more similar to each other.
Can I do some adjustment to kMeans or maybe you know another algorithm in sklearn that can do that?
What it looks like you are wanting is a clustering on 1-dimensional data. One way to to work this out is to use the Jenks Natural Breaks (google it to get an explanation of it).
I didn't write this function (lots of credit goes to @Frank with his solution here)
Given your dataframe:
import pandas as pd
df = pd.DataFrame([
['russian', 0.457039],
['man', 0.286875],
['woman', 0.129939],
['bit', 0.092721],
['write', 0.065424],
['age', 0.064347],
['escap', 0.062675],
['game', 0.062606]], columns = ['node','bc'])
Code with Jenks Natural Break Function:
def get_jenks_breaks(data_list, number_class):
data_list.sort()
mat1 = []
for i in range(len(data_list) + 1):
temp = []
for j in range(number_class + 1):
temp.append(0)
mat1.append(temp)
mat2 = []
for i in range(len(data_list) + 1):
temp = []
for j in range(number_class + 1):
temp.append(0)
mat2.append(temp)
for i in range(1, number_class + 1):
mat1[1][i] = 1
mat2[1][i] = 0
for j in range(2, len(data_list) + 1):
mat2[j][i] = float('inf')
v = 0.0
for l in range(2, len(data_list) + 1):
s1 = 0.0
s2 = 0.0
w = 0.0
for m in range(1, l + 1):
i3 = l - m + 1
val = float(data_list[i3 - 1])
s2 += val * val
s1 += val
w += 1
v = s2 - (s1 * s1) / w
i4 = i3 - 1
if i4 != 0:
for j in range(2, number_class + 1):
if mat2[l][j] >= (v + mat2[i4][j - 1]):
mat1[l][j] = i3
mat2[l][j] = v + mat2[i4][j - 1]
mat1[l][1] = 1
mat2[l][1] = v
k = len(data_list)
kclass = []
for i in range(number_class + 1):
kclass.append(min(data_list))
kclass[number_class] = float(data_list[len(data_list) - 1])
count_num = number_class
while count_num >= 2: # print "rank = " + str(mat1[k][count_num])
idx = int((mat1[k][count_num]) - 2)
# print "val = " + str(data_list[idx])
kclass[count_num - 1] = data_list[idx]
k = int((mat1[k][count_num] - 1))
count_num -= 1
return kclass
# Get values to find the natural breaks
x = list(df['bc'])
# Calculate the break values.
# I want 2 groups, so parameter is 2.
# If you print (get_jenks_breaks(x, 2)), it will give you 3 values: [min, break1, max]
# Obviously if you want more groups, you'll need to adjust this and also adjust the assign_cluster function below.
breaking_point = get_jenks_breaks(x, 2)[1]
# Creating group for the bc column
def assign_cluster(bc):
if bc < breaking_point:
return 0
else:
return 1
# Apply `assign_cluster` to `df['bc']`
df['cluster'] = df['bc'].apply(assign_cluster)
Output:
print (df)
node bc cluster
0 russian 0.457039 1
1 man 0.286875 1
2 woman 0.129939 1
3 bit 0.092721 0
4 write 0.065424 0
5 age 0.064347 0
6 escap 0.062675 0
7 game 0.062606 0