Search code examples
rstatisticsnormal-distribution

Interpreting Shapiro Wilk Test in R

Hello statisticians and data enthusiasts !!

I am working on a dataset to apply linear discriminant analysis on it. And i'm having trouble to finding out columns with good enough normal distributions score of shapiro.test, because all the p-values are up-to the mark of 0.05.

Below is the shapiro.test results on my Data

              statistic  p.value     
Id            0.9548438  7.972013e-21
MSSubClass    0.8045693  9.108194e-39
LotFrontage   0.8804029  2.001693e-29
LotArea       0.3510589  7.933654e-58
OverallQual   0.9480078  2.686457e-22
OverallCond   0.8289229  6.774229e-37
YearBuilt     0.9255974  2.77022e-26 
YearRemodAdd  0.8628004  6.72028e-34 
MasVnrArea    0.639286   6.556645e-48
BsmtFinSF1    0.8479603  2.813854e-35
BsmtFinSF2    0.3272826  1.850254e-58
BsmtUnfSF     0.9304219  1.639911e-25
TotalBsmtSF   0.917352   1.611332e-27
X1stFlrSF     0.9269462  4.513223e-26
X2ndFlrSF     0.7668042  2.514882e-41
LowQualFinSF  0.09799004 9.589248e-64
GrLivArea     0.9279825  6.597611e-26
BsmtFullBath  0.6582952  3.760666e-47
BsmtHalfBath  0.2429119  1.466616e-60
FullBath      0.7193559  4.231488e-44
HalfBath      0.6380019  4.581582e-48
BedroomAbvGr  0.849803   4.115551e-35
KitchenAbvGr  0.2197959  4.221203e-61
TotRmsAbvGrd  0.9422768  2.004964e-23
Fireplaces    0.7552301  4.83098e-42 
GarageYrBlt   0.9209432  2.816783e-26
GarageCars    0.8353703  2.301685e-36
GarageArea    0.9753273  4.016963e-15
WoodDeckSF    0.7685159  3.227985e-41
OpenPorchSF   0.7271672  1.135905e-43
EnclosedPorch 0.4144382  4.849485e-56
X3SsnPorch    0.09493385 8.307268e-64
ScreenPorch   0.2982077  3.305688e-59
PoolArea      0.04120243 7.111538e-65
MiscVal       0.05823268 1.529907e-64
MoSold        0.968784   3.178973e-17
YrSold        0.8970975  3.420194e-30
SalePrice     0.8696715  3.206142e-33

Histogram of all desired columns

enter image description here

But i'm having trouble interpreting these results, as i'm new to stats and R language.

Kindly guide to interpret this accurately in order to find Normally distributed columns.

Solution

  • in order to understand the p-value you have to understand what the corresponding statistical test is actually testing.

    In case of the Shapiro-Wilk Normality Test the null hypothesis is the underlying data has a normal distribution. The p-value then measures (more or less) how likely this is. Often we accept the null hypothesis if the p-value is greater or equal than 0.05. This means that in only 5% of the cases we reject the null hypothesis although it would be correct (Type I error).

    In your case none of the p-values is anywhere near to be accepted. And a brief look at the histograms reveals that indeed none of the variables seem to have a normal distribution. A normal distribution would rather look like this:enter image description here

    The histogram should be symmetric and bell-shaped. Hope this helps.