Problem with arithmetic using logarithms to avoid numerical underflow (take 2)
Having seen the above and having seen softmax normalization I was trying to normalize a vector while avoiding overflow -
that is if I have an array
x[1], x[2] x[3], x[4], ... , x[n]
the normalized form for me has the sum of squares of elements as 1.0
and is obtained by dividing each element by
sqrt(x[1]*x[1]+x[2]*x[2]+...+x[n]*x[n])
now the sum of squares can overflow even if the square root is small enough to fit into a floating point variable, so I imagined one could do something like
s=(2*log(fabs(x[1]))+2*log(fabs(x[2]))+...+2*log(fabs(x[n])))/2
and calculating the elements as
exp(log(fabs(x[1]))-s), ..., exp(log(fabs(x[n]))-s
BUT
The above is incorrect as log(A+B) is not log(A)+log(B) - now is there a way to do vector normalization that avoids overflow better?
You seem to be making the assumption that:
log(x^2 + y^2)
is the same as:
log(x^2) + log(y^2)
However, this isn't correct, as you can't simplify the logarithm of a sum like that.