I followed a tutorial that shows how to do tensors operations in a correct way. They said that usually the operations between tensors are done manually via loops of iterating through the tensor array. Then they showed the better way, by doing 'dot product' with triangular unit matrix, to do averaging to the tensors without losing its spatial information and shows that both methods produces the same results with print(torch.allclose(xbow, xbow2)) which gave 'True' as the return value to show both methods works the same way. But when I followed their path, my results ended up as 'False', showing a probable differences in tensor operations result.
From what I asked to nearby expert around me, since they use a random number generator torch.randn() as a way to produce tensors, the number produced could gave different floating points precision thus reducing the accuracy. Although they are sure about it, they don't know why the tutorials doesn't face the same problem as I get. So going through with what I have, I use set_printoptions(precision=1) to limit tensor value precision point. But the results are still 'False'. What did I do wrong or what should I look for here?
Codes
Ways of producing tensors from tutorials
torch.manual_seed(1337)
B,T,C = 4,8,2 # batch, time, channels
x = torch.randn(B,T,C)
print(x.shape)
print(x[0])
it shows
torch.Size([4, 8, 2])
tensor([[ 0.2, -0.1],
[-0.4, -0.9],
[ 0.6, 0.0],
[ 1.0, 0.1],
[ 0.4, 1.2],
[-1.3, -0.5],
[ 0.2, -0.2],
[-0.9, 1.5]])
To do the averaging with loops from tutorials
# We want x[b, t] = mean_{i<=t} x[b,i]
xbow = torch.zeros((B,T,C))
for b in range(B):
for t in range(T):
xprev = x[b, :t+1] # (t,C)
xbow[b,t] = torch.mean(xprev, 0)
I try printing the first member of the tensors
xbow[0]
with results
tensor([[ 0.2, -0.1],
[-0.1, -0.5],
[ 0.1, -0.3],
[ 0.4, -0.2],
[ 0.4, 0.1],
[ 0.1, -0.0],
[ 0.1, -0.1],
[-0.0, 0.1]])
Then the tutorial show different method, with torch.tril(torch.ones(T,T))
wei = torch.tril(torch.ones(T,T))
wei = wei / wei.sum(1, keepdim=True)
xbow2 = wei @ x # B(T, T, T) @ (B, T, C ) ----> (B, T, C)
Then I print the first member of the result
print(xbow2[0])
and the result is
tensor([[ 0.2, -0.1],
[-0.1, -0.5],
[ 0.1, -0.3],
[ 0.4, -0.2],
[ 0.4, 0.1],
[ 0.1, -0.0],
[ 0.1, -0.1],
[-0.0, 0.1]])
It looks equal from the first member, but when I do xbow == xbow2 it shows that some of the tensors aren't equal
tensor([[[ True, True],
[ True, True],
[ True, False],
[ True, True],
[ True, False],
[False, True],
[False, False],
[ True, True]],
[[ True, True],
[ True, True],
[False, True],
[ True, True],
[False, False],
[False, False],
[False, True],
[False, True]],
[[ True, True],
[ True, True],
[False, False],
[ True, True],
[False, False],
[False, True],
[False, False],
...
What happened here?
Edits 2: What I want here to have precise floating value in both method of calculation. I know that I can compare both with methods here floating-point-is-hard blog by Bruce Dawson and I can check that these values are equal with torch.allclose(xbow,xbow2) but is there any way that the precision value stays thus reducing some probable information loss in the future? Like for example, when I want to average images and compress them, is there any possibility these floating points fluctuation become a hassle? How to avoid that?
You're misunderstanding what set_printoptions does. set_printoptions toggles how many decimal places are shown when printing a tensor. It's purely for visual clarity - the underlying tensor itself is not changed.
For comparing xbow and xbow2, you need to adjust the tolerance values on torch.allclose. The error between the two tensors is just outside the default value.
torch.allclose(xbow, xbow2)
> False
torch.allclose(xbow, xbow2, atol=1e-7)
> True
If you want to compare the tensors at lower precision, you can use torch.round to limit the decimal places. You can also use this to find the decimal place where the error exceeds the default tolerance.
torch.allclose(
torch.round(xbow, decimals=7),
torch.round(xbow2, decimals=7)
)
> False
torch.allclose(
torch.round(xbow, decimals=6),
torch.round(xbow2, decimals=6)
)
> True
You can also compute the values with 64 bit floats for higher precision
torch.manual_seed(1337)
B,T,C = 4,8,2 # batch, time, channels
x = torch.randn(B,T,C, dtype=torch.float64)
xbow = torch.zeros((B,T,C), dtype=torch.float64)
for b in range(B):
for t in range(T):
xprev = x[b, :t+1] # (t,C)
xbow[b,t] = torch.mean(xprev, 0)
wei = torch.tril(torch.ones(T,T, dtype=torch.float64))
wei = wei / wei.sum(1, keepdim=True)
xbow2 = wei @ x
torch.allclose(xbow, xbow2)
> True