How to profile GPflow optimization process using timeline?

Question

I am trying to profile GPflow using timeline and visualizing it with chrome tracing. But the trace does not seem to show the optimization process (only model construction and prediction). I define a custom config:

custom_config = gpflow.settings.get_settings()
custom_config.profiling.output_file_name = 'gpflow_timeline'
custom_config.profiling.dump_timeline = True

And try to make a simple prediction after optimization:

with gpflow.settings.temp_settings(custom_config), gpflow.session_manager.get_session().as_default():
   k = gpflow.kernels.RBF()
   m = gpflow.models.GPR(X_train, y_train, kern=k)
   run_adam(m, lr=0.1, iterations=100, callback=__PrintAction(m, 'GPR with Adam'))
   mean, var = m.predict_y(X_test)

where Adam optimizer is defined as:

class __PrintAction(Action):
   def __init__(self, model, text):
       self.model = model
       self.text = text

   def run(self, ctx):
       likelihood = ctx.session.run(self.model.likelihood_tensor)
       print('{}: iteration {} likelihood {:.4f}'.format(self.text, ctx.iteration, likelihood))

def run_adam(model, lr, iterations, callback=None):
   adam = gpflow.train.AdamOptimizer(lr).make_optimize_action(model)
   actions = [adam] if callback is None else [adam, callback]
   loop = Loop(actions, stop=iterations)()
   model.anchor(model.enquire_session()) 

Is it somehow possible to also show the optimization trace on the timeline?

Answer 1

Extension to @tadejk answer:

You can modify gpflowrc in GPflow/gpflow project folder instead or create it in the same folder where you run the code and tune your profiling parameters there.

[logging]
# possible levels: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET
level = WARNING

[verbosity]
tf_compile_verb = False

[dtypes]
float_type = float64
int_type = int32

[numerics]
jitter_level = 1e-6
# quadrature can be set to: allow, warn, error
ekern_quadrature = warn

[profiling]
dump_timeline = False
dump_tensorboard = False
output_file_name = timeline
output_directory = ./
each_time = False

[session]
intra_op_parallelism_threads = 0
inter_op_parallelism_threads = 0

Not 100% sure, but merging everything into one json file might be a bad idea. Single file produced by a session.run, therefore merging everything into one can mess things up.