I used GPU profiling functionality in tensorflow with code like this:
run_opts = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
meta = tf.RunMetadata()
sess.run(op, options=run_opts, run_metadata=meta)
writer.add_run_metadata(meta, 'profile')
writer.flush()
Unfortunately, when I load it in the tensorboard, I see a lot of empty boxes like on the attached image.
I found out what was wrong. In the left pane you should choose device. Then you should choose gpu:0. It's definitely a usability problem in tensorboard.