I've got a 64-bit ARM machine that I'd like to run 32-bit ARM binaries on. As a test case I've built a small hello world for 32-bit ARM using the arm-linux-gnueabihf-gcc toolchain. file shows it as:
root@ubuntu:/home/ubuntu# file hello
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=61ffe5e22117a6d4c2ae37a1f4c76617d3e5facc, not stripped
But trying to run it produces:
root@ubuntu:/home/ubuntu# ./hello
bash: ./hello: cannot execute binary file: Exec format error
Based on a prior question, I checked whether the kernel was built with the CONFIG_COMPAT option, and it was:
root@ubuntu:/home/ubuntu# grep CONFIG_COMPAT= /boot/config-$(uname -r)
CONFIG_COMPAT=y
I also verified that the armhf architecture has been added and that the armhf version of the loader is present. Note that the loader itself doesn't run either:
root@ubuntu:/home/ubuntu# dpkg --print-foreign-architectures
armhf
root@ubuntu:/home/ubuntu# dpkg -l libc6:armhf
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=============-============-=================================
ii libc6:armhf 2.30-0ubuntu2 armhf GNU C Library: Shared libraries
root@ubuntu:/home/ubuntu# file /lib/arm-linux-gnueabihf/ld-2.30.so
/lib/arm-linux-gnueabihf/ld-2.30.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=dff2b536287d61ddca68f3e001e14b7c235bbf68, stripped
root@ubuntu:/home/ubuntu# /lib/arm-linux-gnueabihf/ld-2.30.so
bash: /lib/arm-linux-gnueabihf/ld-2.30.so: cannot execute binary file: Exec format error
Other relevant system info:
root@ubuntu:/home/ubuntu# cat /proc/cpuinfo
processor : 0
BogoMIPS : 400.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm
CPU implementer : 0x43
CPU architecture: 8
CPU variant : 0x1
CPU part : 0x0af
CPU revision : 2
root@ubuntu:/home/ubuntu# uname -a
Linux ubuntu 5.3.0-24-generic #26-Ubuntu SMP Thu Nov 14 01:14:25 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
I'm running out of things to try at this point. Any idea how to get the kernel to recognize these binaries and run them?
Ok, it looks like the underlying problem here is not in software, unfortunately. The CPU we're using, the Cavium ThunderX2, is one of the few 64-bit ARM chips that does not have aarch32 support. Quoting from WikiChip:
So, this explains why it's not able to run 32-bit ARM binaries. I'm still fairly sure that other 64-bit ARM chips, like the Cortex-A57, are able to do this.
Update: older 32-bit ARM binaries do indeed work on aarch64 with a CPU that supports it, as shown below on an AWS ARM a1.metal instance:
ubuntu@ip-172-31-12-156:~$ cat /proc/cpuinfo | tail
processor : 15
BogoMIPS : 166.66
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3
ubuntu@ip-172-31-12-156:~$ uname -a
Linux ip-172-31-12-156 4.15.0-1054-aws #56-Ubuntu SMP Thu Nov 7 16:18:50 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@ip-172-31-12-156:~$ dpkg --print-foreign-architectures
armhf
ubuntu@ip-172-31-12-156:~$ file hello_hf
hello_hf: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 3.2.0, BuildID[sha1]=c95f0c46dfab925db53506751d7677156e334e5c, not stripped
ubuntu@ip-172-31-12-156:~$ ./hello_hf
hello, world!