I'm looking to use TensorFlow Addons (9.1) with TensorFlow (2.2-stable). There is a function tfa.image.dense_image_warp that I wish to use. However, it uses bilinear interpolation which I'm having trouble understanding if it is deterministic.
According to pull-request, https://github.com/tensorflow/tensorflow/pull/39243, the author of the repo NVIDIA/tensorflow-determinism (Duncan Riach - Thank you again!), there are issues with the bilinear op.
However, as far as I can see from the source code of tfa.image.dense_image_warp, in particular: https://github.com/tensorflow/addons/blob/ad132da23a8162eb97c435676dd7426e622a0074/tensorflow_addons/image/dense_image_warp.py#L25, it seems that the bilinear interp is self-contained.
Does this mean that the tfa.image.dense_image_warp is safe to be deemed GPU-deterministic? Can anyone who is more familiar with this op or GPU-determinism in general confirm?
Edit:
Answer is below. Please see Issue 39751 that I opened. The issue is that tfa.dense_image_warp on backpropagation uses CUDA Atomic addition. As a result, for determinism (and arguably for stable non-deterministic training), Atomic addition is not suitable for this function's backprop.
Solved my issue. Please see Issue 39751. The issue is that tfa.dense_image_warp on backpropagation uses CUDA Atomic addition. As a result, for determinism (and arguably for stable non-deterministic training), Atomic addition is not suitable for this function's backprop.