I'm attempting to run the Image Recognition with the C++ API tutorial and I have an Illegal instruction error, while trying to execute label_image, after the TensorFlow compilation using Bazel.
I did the following steps:
# After installing the bazel dependencies, I get the bazel installer
$ mkdir ~/bazel-download && cd ~/bazel-download
$ wget https://github.com/bazelbuild/bazel/releases/download/0.3.0/bazel-0.3.0-installer-linux-x86_64.sh -O bazel-0.3.0-installer-linux-x86_64.sh
$ chmod +x bazel-0.3.0-installer-linux-x86_64.sh
# Install bazel in ~/bin
$ ./bazel-0.3.0-installer-linux-x86_64.sh --user
# Add bazel to the path, if not done already
$ printf '\nexport PATH=$PATH:"~/bin/"\n' >> ~/.bashrc
# Before this, I create a new terminal to refresh the bash PATH
$ mkdir ~/inceptionV3 && cd ~/inceptionV3
# Get a stable version of TensorFlow
$ git clone https://github.com/tensorflow/tensorflow -b r0.9
$ cd tensorflow
# Add the InceptionV3 data/models for the C++ api
$ wget https://storage.googleapis.com/download.tensorflow.org/models/inception_dec_2015.zip -O tensorflow/examples/label_image/data/inception_dec_2015.zip
$ unzip tensorflow/examples/label_image/data/inception_dec_2015.zip -d tensorflow/examples/label_image/data/
# Configure tensorflow: set python path, no Google Cloud Platform support, no GPU support
$ ./configure
# Run bazel build with the allocated resources
$ bazel build -c opt --copt=-mavx --verbose_failures --local_resources 2048,2.0,1.0 -j 1 tensorflow/examples/label_image/...
# -- Here's the last log output from bazel --
INFO: From Compiling tensorflow/core/common_runtime/function.cc:
tensorflow/core/common_runtime/function.cc: In lambda function:
tensorflow/core/common_runtime/function.cc:392:60: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
} else if (rets->size() != ctx->num_outputs()) {
^
INFO: Elapsed time: 6929.927s, Critical Path: 69.23s
# Look like there's no error during the compilation, but now, if I run the generated executable:
$ ./bazel-bin/tensorflow/examples/label_image/label_image
Illegal instruction
Also, I'm running on Docker with an Ubuntu 14.04.4 LTS x86_64 container (gcc/g++ version is 4.8.4).
I tried running this with an other setup, such as using apt-get install for bazel, but I still get a an Illegal instruction error after running the executable with a new compilation.
That being said, the Python part of the tutorial work fine(using python 2.7.6). Any idea how to solve to issue for the C++ API?
edit1: (add more infos about cpu) Here's the output I get from /proc/cpuinfo.
edit2: (trying to debug tensorflow) Using this command to compile:
$ bazel build -c dbg --strip=always --copt=-mavx --verbose_failures --local_resources 2048,2.0,1.0 -j 1 tensorflow/examples/label_image/...
And I tried to debug with gdb:
$ -q bazel-bin/tensorflow/examples/label_image/label_image
Reading symbols from bazel-bin/tensorflow/examples/label_image/label_image...(no debugging symbols found)...done.
(gdb) set disable-randomization off
(gdb) run
Starting program: /root/.cache/bazel/_bazel_root/b54d699ba1afcab684f4628c78701dbe/execroot/tensorflow/bazel-out/local-dbg/bin/tensorflow/examples/label_image/label_image
During startup program terminated with signal SIGILL, Illegal instruction.
(gdb) backtrace
No stack.
(gdb) handle SIGILL nostop
Signal Stop Print Pass to program Description
SIGILL No Yes Yes Illegal instruction
(gdb) run
Starting program: /root/.cache/bazel/_bazel_root/b54d699ba1afcab684f4628c78701dbe/execroot/tensorflow/bazel-out/local-dbg/bin/tensorflow/examples/label_image/label_image
During startup program terminated with signal SIGILL, Illegal instruction.
(gdb) backtrace
No stack.
(gdb) info files
Symbols from "/root/.cache/bazel/_bazel_root/b54d699ba1afcab684f4628c78701dbe/execroot/tensorflow/bazel-out/local-dbg/bin/tensorflow/examples/label_image/label_image".
Local exec file:
`/root/.cache/bazel/_bazel_root/b54d699ba1afcab684f4628c78701dbe/execroot/tensorflow/bazel-out/local-dbg/bin/tensorflow/examples/label_image/label_image', file type elf64-x86-64.
Entry point: 0x434b10
0x0000000000400270 - 0x000000000040028c is .interp
0x000000000040028c - 0x00000000004002ac is .note.ABI-tag
0x00000000004002ac - 0x00000000004002cc is .note.gnu.build-id
0x00000000004002d0 - 0x0000000000400380 is .gnu.hash
0x0000000000400380 - 0x00000000004027e0 is .dynsym
0x00000000004027e0 - 0x0000000000404667 is .dynstr
0x0000000000404668 - 0x0000000000404970 is .gnu.version
0x0000000000404970 - 0x0000000000404b70 is .gnu.version_r
0x0000000000404b70 - 0x0000000000431360 is .rela.dyn
0x0000000000431360 - 0x00000000004334a8 is .rela.plt
0x00000000004334a8 - 0x00000000004334c2 is .init
0x00000000004334d0 - 0x0000000000434b10 is .plt
0x0000000000434b10 - 0x00000000027cfe2f is .text
0x00000000027cfe30 - 0x00000000027cfe39 is .fini
0x00000000027cfe40 - 0x0000000003890ed0 is .rodata
0x0000000003890ed0 - 0x0000000003acc1ec is .eh_frame_hdr
0x0000000003acc1f0 - 0x000000000441fc2c is .eh_frame
0x000000000441fc2c - 0x000000000444474f is .gcc_except_table
0x0000000004644dd0 - 0x0000000004644de0 is .tdata
0x0000000004644de0 - 0x0000000004644df8 is .tbss
0x0000000004644de0 - 0x0000000004645a70 is .init_array
0x0000000004645a70 - 0x0000000004645a78 is .fini_array
0x0000000004645a78 - 0x0000000004645a80 is .jcr
0x0000000004645a80 - 0x00000000046a5d50 is .data.rel.ro
0x00000000046a5d50 - 0x00000000046a5f90 is .dynamic
0x00000000046a5f90 - 0x00000000046a6000 is .got
0x00000000046a6000 - 0x00000000046a6b30 is .got.plt
0x00000000046a6b40 - 0x00000000046a70d0 is .data
0x00000000046a70e0 - 0x00000000046aae18 is .bss
(gdb) break main
Breakpoint 1 at 0x436cc0
(gdb) run
Starting program: /root/.cache/bazel/_bazel_root/b54d699ba1afcab684f4628c78701dbe/execroot/tensorflow/bazel-out/local-dbg/bin/tensorflow/examples/label_image/label_image
During startup program terminated with signal SIGILL, Illegal instruction.
(gdb) backtrace
No stack.
So far, since the Illegal instruction error is caused by a SIGILL signal, then I guess my current architecture doesn't match the generated machine code. Yet, I'm not sure how to deal with this particular issue.
After a couple of search, it seem --copt=-mavx is actualy a parameter passed to gcc in order to optimize the architecture on an OSX machine as pointed over here. So there's no way it could works on my linux "PC" machine.