What is difference between PCA , TruncatedSVD and ICA in details?

Question

Can anybody tell me the differences between PCA(Principal component analysis ) , TruncatedSVD(Truncated singular value decomposition) and ICA(Independent component analysis) in detail?

Answer 1

Doing it in detail will require long pages PDF document :-).

But the idea is simple:

1. Principal Component Analysis (PCA) - analyze the data native coordinates. Namely the coordinated which along the data has its most energy (Variance). For n Samples of dimension d there will be $ d $ orthogonal directions. Namely the data projected on them has no correlation. If we look on the data as Random Variables, it means we found a coordinate system where the Cross Correlation (First Moment) of any pair from the projected data is vanished.
   This is a very efficient way to approximate the data in lower dimensionality by keeping most of its energy.
2. Truncated SVD - One could show that one of the ways of calculating those system of coordinate is using the SVD. Hence this is method to apply the ideas behind PCA.
3. Independent Component Analysis (ICA) - This is one step farther from PCA. While in PCA we dealt with only First Order Moments of the data (Correlation) in ICA we're looking into higher moments and try to find a projection of data which vanishes higher moments (Think of Lack of Correlation vs. Probability Independence).