Search code examples
pythontensorflowobject-detection

Is there a way to implement a custom learning rate in Tensorflow Object Detection?

I am working on an object detection problem and I'd like to use the Cyclical Learning Rate. Thing is, this specific learning does not exist in the protos of Tensorflow Object Detection. I'd like to know if it is possible for me to modify the protos (or another file) as to implement this new learning rate method?

I am using Tensorflow 1.14 with the latest updated version of the Tensorflow Object Detection repo.

I tried to modify the optimizer.proto and optimizer_pb2.py files. I am only showing the sections I modified.

optimizer.proto

// Configuration message for optimizer learning rate.
message LearningRate {
  oneof learning_rate {
    ConstantLearningRate constant_learning_rate = 1;
    ExponentialDecayLearningRate exponential_decay_learning_rate = 2;
    ManualStepLearningRate manual_step_learning_rate = 3;
    CosineDecayLearningRate cosine_decay_learning_rate = 4;
    CosineDecayRestartLearningRate cosine_decay_restart_learning_rate = 5; // Added
  }
}
...
// Added for test
message CosineDecayRestartLearningRate {
  optional uint32 total_steps = 1 [default = 400000];
}

optimizer_pb2.py

_LEARNINGRATE = _descriptor.Descriptor(
...
full_name='object_detection.protos.LearningRate.cosine_decay_restart_learning_rate', index=4,
      number=5, type=11, cpp_type=10, label=1,
      has_default_value=False, default_value=None,
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      options=None),
...)
...
_COSINEDECAYRESTARTLEARNINGRATE = _descriptor.Descriptor(
  name='CosineDecayRestartLearningRate',
  full_name='object_detection.protos.CosineDecayRestartLearningRate',
  filename=None,
  file=DESCRIPTOR,
  containing_type=None,
  fields=[
    _descriptor.FieldDescriptor(
      name='total_steps', full_name='object_detection.protos.CosineDecayLearningRate.total_steps', index=1,
      number=2, type=13, cpp_type=3, label=1,
      has_default_value=True, default_value=4000000,
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      options=None),
  ],
  extensions=[
  ],
  nested_types=[],
  enum_types=[
  ],
  options=None,
  is_extendable=False,
  syntax='proto2',
  extension_ranges=[],
  oneofs=[
  ],
  serialized_start=1671,
  serialized_end=1861,
)
_LEARNINGRATE.fields_by_name['cosine_decay_restart_learning_rate'].message_type = _COSINEDECAYRESTARTLEARNINGRATE
_LEARNINGRATE.oneofs_by_name['learning_rate'].fields.append(
  _LEARNINGRATE.fields_by_name['cosine_decay_restart_learning_rate'])
_LEARNINGRATE.fields_by_name['cosine_decay_restart_learning_rate'].containing_oneof = _LEARNINGRATE.oneofs_by_name['learning_rate']
DESCRIPTOR.message_types_by_name['CosineDecayRestartLearningRate'] = _COSINEDECAYRESTARTLEARNINGRATE
CosineDecayRestartLearningRate = _reflection.GeneratedProtocolMessageType('CosineDecayRestartLearningRate', (_message.Message,), dict(
  DESCRIPTOR = _COSINEDECAYRESTARTLEARNINGRATE,
  __module__ = 'object_detection.protos.optimizer_pb2'
  # @@protoc_insertion_point(class_scope:object_detection.protos.CosineDecayRestartLearningRate)
  ))
_sym_db.RegisterMessage(CosineDecayRestartLearningRate)

I didn't expect it to work since at no point did I reach a step where I add to input code for the learning rate, and it does give me an error.

  File "/home/renart/Tensorflow/models/research/object_detection/protos/optimizer_pb2.py", line 253, in <module>
    options=None),
  File "/home/renart/Tensorflow/venv-1.13/lib/python3.5/site-packages/google/protobuf/descriptor.py", line 534, in __new__
    return _message.default_pool.FindFieldByName(full_name)
KeyError: "Couldn't find field object_detection.protos.LearningRate.cosine_decay_restart_learning_rate"

Unfortunately, the error comes from Protobuf and not Tensorflow, so I still have no idea where to look in order to implement the learning rate.

Solution

  • I do not think you should modify optimizer_pb2.py manually. It is generated by protoc command. The exact step is in here https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md#protobuf-compilation

    You should only modify optimizer.proto and follow the above instruction to recompile the .proto files.

    To add a new learning rate schedule, you may consider add it in this file https://github.com/tensorflow/models/blob/master/research/object_detection/utils/learning_schedules.py since it is where the official learning rate schedules reside.

    The configurations for learning rate is interpreted in this file: https://github.com/tensorflow/models/blob/0b3a8abf095cb8866ca74c2e118c1894c0e6f947/research/object_detection/builders/optimizer_builder.py#L103 You should call build your learning rate schedule here.