I want to use PCA algorithm from scratch, but i want to reduce 784 features to about for example 70 features not 2. what i've tried before is this code below. In this code for part "Choosing k eigenvectors with the largest eigenvalues" how can i choose k?
import numpy as np
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
import pandas as pd
data_train = pd.read_csv('trainData.csv',header=None)
label_train = pd.read_csv('trainLabels.csv',header=None)
data_train = StandardScaler().fit_transform(data_train)
# OR we can do this with one line of numpy for COV:
cov_mat = np.cov(data_train.T)
# Compute the eigen values and vectors using numpy
eig_vals, eig_vecs = np.linalg.eig(cov_mat)
# Make a list of (eigenvalue, eigenvector) tuples
eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]
# Sort the (eigenvalue, eigenvector) tuples from high to low
eig_pairs.sort(key=lambda x: x[0], reverse=True)
for i in eig_pairs:
print(i[0])
#Choosing k eigenvectors with the largest eigenvalues
matrix_w = np.hstack((eig_pairs[0][1].reshape(784,1), eig_pairs[1][1].reshape(784,1)))
print('Matrix W:\n', matrix_w)
transformed = matrix_w.T.dot(data_train.T)
Take a look at how matrix_w is created. The h_stack function takes a tuple of arrays and stacks them horizontally. What you want to do is create a tuple that contains the eigenvectors of the k largest eigenvalues and create the matrix:
eigenvectors = tuple([eig_pairs[i][1].reshape(784,1) for i in range(k)])
matrix_w = np.hstack(eigenvectors)