Search code examples
pythonanacondacondaprotobuf-c

How can I replace the conda package for libprotobuf with the cloned GitHub version within a conda environment

I am trying to install caffe within a conda environment. Caffe needs Google's protobuf package. I had already git clone'd protobuf and have it within my \usr directories. However, when I try to install caffe within the conda environment, the version of libprotobuf installed doesn't correctly convert the proto file to c++ code.

Consider the following code:

syntax = "proto2";

package test1;


message Datum {
  optional int32 channels = 1;
}

When I attempt to translate it from within my base environment, everything is fine:

(base) me@balin:~/Projects/caffe$ make clean
(base) me@balin:~/Projects/caffe$ make superclean
Deleting the following generated files:
./temp5.pb.cc
./temp5.pb.h
(base) me@balin:~/Projects/caffe$ protoc --cpp_out=. temp5.proto
(base) me@balin:~/Projects/caffe$ g++ temp5.pb.cc -c
(base) me@balin:~/Projects/caffe$

However, when I try the same thing in the environment I want to use for caffe, I get this result:

(dnn_track5) me@balin:~/Projects/caffe$ make clean
(dnn_track5) me@balin:~/Projects/caffe$ make superclean
Deleting the following generated files:
./temp5.pb.cc
(dnn_track5) me@balin:~/Projects/caffe$ protoc --cpp_out=. temp5.proto
(dnn_track5) me@balin:~/Projects/caffe$ g++ temp5.pb.cc -c
temp5.pb.cc: In member function ‘virtual const char* test1::Datum::_InternalParse(const char*, google::protobuf::internal::ParseContext*)’:
temp5.pb.cc:150:58: error: ‘ReadVarint’ is not a member of ‘google::protobuf::internal’
           channels_ = ::PROTOBUF_NAMESPACE_ID::internal::ReadVarint(&ptr);
                                                          ^~~~~~~~~~
temp5.pb.cc:150:58: note: suggested alternative: ‘ReadVarint32’
           channels_ = ::PROTOBUF_NAMESPACE_ID::internal::ReadVarint(&ptr);
                                                          ^~~~~~~~~~
                                                          ReadVarint32

All I can think to do is replace each file within the ~\anaconda2\envs\dnn_track5 sub-directories that were installed by conda with the files installed when I built protobuf from the GitHub clone. Am I right in this (I doubt it).

How can I create a conda environment where I can use caffe and still have a working protobuf?

Solution

  • There is no direct way to use conda to install directly from a github repo, so it is worth to look at why your code is not working in your caffe env.

    As of this commit(From October 2019) there is a distinction between 

    ::PROTOBUF_NAMESPACE_ID::internal::ReadVarint64

    and

    ::PROTOBUF_NAMESPACE_ID::internal::ReadVarint32

    When you conda install caffe, it downloads(at least for me) libprotobuf-3.11.2, which is the most recent version from December 2019, so the version downloaded as a dependency for caffe is actually more recent than what you are trying to use in your code.

    You have several options now:

    1. Request a specific version of protobuf that has the API you want to use. I.e. 3.10.0, which is from 3rd October 2019:

      conda create -n caffe -c conda-forge python=3.7 caffe libprotobuf=3.10.0

    2. Create a custom conda channel and put your own libprotobuf.tar.bz2 in there

    3. Adjust your code to use the most recent libprotobuf API