Is there a resource for how and if they do it exactly? I'm curious if dependencies are recognized in the graph and parallelized accordingly.
https://www.tensorflow.org/tutorials/images/deep_cnn
https://www.tensorflow.org/deploy/distributed
https://www.tensorflow.org/guide/faq
https://github.com/google/seq2seq/issues/44
These aren't clear to me because they talk about doing it manually but don't seem to explain how or if the graph is parallelized automatically.
The reason is that TensorFlow does not supply an automatic mechanism for distributing a graph over multiple execution units, it is assumed that the developers do that themselves. However, it obviously keeps track of all the dependencies in the graph so that if you do it manually, it will make sure things are computed in the right order.
If you would like to get out of the box distributed/multi-GPU capabilities, take a look at tensor pack: https://github.com/tensorpack/tensorpack