Let's take two sets:
a = [5,5,5,5,5,4,4,4,4,3,3,3,2,2,1]
b = [5,4,3,2,1]
We perform the KS-Test using Python:
from scipy import stats
stats.ks_2samp(b,a)
KstestResult(statistic=0.2, pvalue=0.9979360165118678, statistic_location=2, statistic_sign=1)
Why is the result a p-value of 0.9979? This means that the distribution of the values in the two sets is almost identical. But it's not! What do I missunderstand?
Kind regards.
The observed value of the KS test statistic, namely 0.2, is actually relatively small, considering the distribution of the test statistic for a reasonable null hypothesis; I think this is where the surprise is coming from.
As mentioned, the usual KS test assumes there are no ties, so we'll have to compute the p-value ourselves. We can make progress by assuming the null hypothesis is that both samples come from a uniform distribution and estimating the p-value by random sampling. (This null hypothesis is more restrictive than the conventional one which just assume the same distribution, not necessarily uniform.)
Here are a few lines of R code to estimate the p-value. In the interest of brevity, it's specific to the problem as stated: a sample of size 5 and a sample of size 15, each one from a uniform distribution on the set { 1, 2, 3, 4, 5 }, and the observed KS test statistic is 0.2.
my.ecdf <- function (x) cumsum (sapply (1:5, function (k) sum (x == k))/length(x))
R <- function (n) sample.int (5, size = n, replace = T)
generate.ks.test.statistic <- function (n) sapply (1:n, function (k) max (abs (my.ecdf (R (15)) - my.ecdf (R (5)))))
ks <- generate.ks.test.statistic (10000)
sum (ks >= 0.2)/10000
For this last input, I get 0.8243. That's not as extreme as the value you reported (more than 0.99), but still enough to show that 0.2 is actually relatively small. You can look at hist(ks) to see what the distribution looks like.