I am trying to compute the L2 norm between two tensors as part of a loss function, but somehow my loss ends up being NaN and I suspect it it because of the way the L2 norm is computed. Can you please explain me the difference between torch.linalg.vector_norm(x-y) and torch.norm(x-y) functions? I think that the x-y difference is getting to small and I want to make sure that these functions handle this case properly.
As mentioned in the comments, torch.norm is depreciated in favor of more explicit functions torch.linalg.vector_norm, torch.linalg.matrix_norm and torch.linalg.norm. This is to make more explicit how the different operations are flattening or preserving dimensions of the input. You can read the documentation (link, link, link) for more details.
For your question about the NaN, I doubt that is being caused by the norm operation. When the x-y difference is small, both torch.linalg.vector_norm(x-y) and torch.norm(x-y) will round to zero instead of producing a NaN.
x = torch.randn(64)
for scale in range(10):
y = x - (10**(-scale)) * x
print(torch.linalg.vector_norm(x-y))
> tensor(7.8801)
> tensor(0.7880)
> tensor(0.0788)
> tensor(0.0079)
> tensor(0.0008)
> tensor(7.8811e-05)
> tensor(7.8175e-06)
> tensor(7.6573e-07)
> tensor(0.)
> tensor(0.)
If you are doing something like dividing a value by the result of the norm operation, that may produce a NaN from division by zero.