Bounding hyperparameter optimization with Tensorflow bijector chain in GPflow 2.0

Question

While doing GP regression in GPflow 2.0, I want to set hard bounds on lengthscale (i.e. limiting lengthscale optimization range). Following this thread (Setting hyperparameter optimization bounds in GPflow 2.0), I constructed a TensorFlow Bijector chain (see bounded_lengthscale function below). However, the bijector chain below does not prevent the model from optimizing outside the supposed bounds. What do I need to change to make the bounded_lengthscale function put hard bounds on optimization?

Below is the MRE:

import gpflow 
import numpy as np
from gpflow.utilities import print_summary
import tensorflow as tf
from tensorflow_probability import bijectors as tfb

# Noisy training data
noise = 0.3
X = np.arange(-3, 4, 1).reshape(-1, 1).astype('float64')
Y = (np.sin(X) + noise * np.random.randn(*X.shape)).reshape(-1,1)

def bounded_lengthscale(low, high, lengthscale):
    """Returns lengthscale Parameter with optimization bounds."""
    affine = tfb.AffineScalar(shift=low, scale=high-low)
    sigmoid = tfb.Sigmoid()
    logistic = tfb.Chain([affine, sigmoid])
    parameter = gpflow.Parameter(lengthscale, transform=logistic, dtype=tf.float32)
    parameter = tf.cast(parameter, dtype=tf.float64)
    return parameter

# build GPR model
k = gpflow.kernels.Matern52()
m = gpflow.models.GPR(data=(X, Y), kernel=k)

m.kernel.lengthscale.assign(bounded_lengthscale(0, 1, 0.5))

print_summary(m)

# train model
@tf.function(autograph=False)
def objective_closure():
    return - m.log_marginal_likelihood()

opt = gpflow.optimizers.Scipy()
opt_logs = opt.minimize(objective_closure,
                        m.trainable_variables)
print_summary(m)

Thanks!

Answer 1

tfb.Sigmoid now accepts low and high parameters, as @Brian Patton forecasted in a comment.

Therefore, the code can be simplified to :

from tensorflow_probability import bijectors as tfb

def bounded_lengthscale(low, high, lengthscale):
    """Make lengthscale tfp Parameter with optimization bounds."""
    sigmoid = tfb.Sigmoid(low, high)
    parameter = gpflow.Parameter(lengthscale, transform=sigmoid, dtype='float32')
    return parameter

m.kernel.lengthscale = bounded_lengthscale(0, 1, 0.5)