Compiling openssl-fips on Solaris10 x86

I am currently trying to compile openssl-fips on a Solaris 10x86 machine, and I am having trouble linking to the fipscanister.o in my use of the library. Here is the problem that shows up when I attempt to link:

{PWD}/dist/openssl/bin/../lib/fipscanister.o: wrong ELF class: ELFCLASS64

From what I have gathered when researching, the problem is this: the fipscanister.o was compiled as a 64 bit binary and it is linking to a 32 bit library. I have tried tracking the problem to my compilation of the openssl-fips. Here is the command that I believe is causing the issue:

./config fipscanisterbuild

This is the part of the output that I believe is messing it up:

Configuring for solaris64-x86_64-gcc
    no-camellia     [default]  OPENSSL_NO_CAMELLIA (skip dir)
    no-gmp          [default]  OPENSSL_NO_GMP (skip dir)
    no-krb5         [krb5-flavor not specified] OPENSSL_NO_KRB5
    no-mdc2         [default]  OPENSSL_NO_MDC2 (skip dir)
    no-rc5          [default]  OPENSSL_NO_RC5 (skip dir)
    no-rfc3779      [default]  OPENSSL_NO_RFC3779 (skip dir)
    no-seed         [default]  OPENSSL_NO_SEED (skip dir)
    no-zlib         [default]
    no-zlib-dynamic [default]
IsMK1MF=0
CC            =gcc
CFLAG         =-fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN   
               -DHAVE_DLFCN_H -m64 -O3 -Wall -DL_ENDIAN -DMD32_REG_T=int
               -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM
EX_LIBS       =-lsocket -lnsl -ldl 
CPUID_OBJ     =x86_64cpuid.o
BN_ASM        =x86_64-gcc.o x86_64-mont.o
DES_ENC       =des_enc.o fcrypt_b.o
AES_ASM_OBJ   =aes-x86_64.o
BF_ENC        =bf_enc.o
CAST_ENC      =c_enc.o
RC4_ENC       =rc4-x86_64.o
RC5_ENC       =rc5_enc.o
MD5_OBJ_ASM   =md5-x86_64.o
SHA1_OBJ_ASM  =sha1-x86_64.o sha256-x86_64.o sha512-x86_64.o
RMD160_OBJ_ASM=
PROCESSOR     =
RANLIB        =/usr/ccs/bin/ranlib
ARFLAGS       =
PERL          =/opt/csw/bin/perl
SIXTY_FOUR_BIT_LONG mode
DES_UNROLL used
DES_INT used
RC4_CHUNK is unsigned long
BF_PTR2 used
e_os2.h => include/openssl/e_os2.h

There are several things that I notice here:

fipscanisterbuild is autodetecting (incorrectly) that I am working on a solaris64-x86_64. Here is my uname -a:
```
SunOS hostname 5.10 Generic_147441-01 i86pc i386 i86pc
```
CFLAG has the -m64 in it. This is forcing the fipscanister.o to compile as a 64 bit binary
At the end it says SIXTY_FOUR_BIT_LONG mode

I have tried to force it to compile as a 32 bit binary with the following methods:

I first tried to use an env variable to force it to compile as a 32 bit binary with the -m32 flag
```
export CC="gcc -m32"
```
I also tried to make the CFLAG use my own env variable with the command
```
export CFLAG="gcc -m32"
```
but to no avail.
I tried to make the configure use my architecture by changing the command
```
./config fipscanisterbuild
```
to
```
./Configure solaris-x86-gcc
```
and it seemed like it worked, at least it changed SIXTY_FOUR_BIT_LONG mode to THIRTY_TWO_BIT mode. However when I compiled and tried to link to the fipscanister.o I got the same ELFCLASS64 error.

A side questions:

Why is my fipscanister option in ./config fipscanister recognizing my machine as an x86_64 when it is a 32 bit machine?

I am extremely stuck so it would be great if anyone could shed light on this very baffling problem.

Solution

After some more knob turning the the environmental variables, I was able to get the triplet solaris-whatever-gcc with 32-bit mode (THIRTY_TWO_BIT mode).

I'm not sure its expected, and it looks more like a bug in OpenSSL's configure system (especially the SYSTEM="gcc" part).

You still might need the CC=gcc -m32 trick.

export KERNEL_BITS=32
export ARCH=x86

export MACHINE="solaris"
export RELAEASE="10"
export SYSTEM="gcc"
export BUILD="Solaris; Solaris 10; x86"

openssl-fips-2.0.5$ ./config 
Operating system: solaris-whatever-gcc
Auto Configuring fipsonly
Auto Configuring fipsonly
Configuring for gcc
Auto Configuring fipsonly
Configuring for gcc
    no-bf           [option]   OPENSSL_NO_BF (skip dir)
    no-camellia     [option]   OPENSSL_NO_CAMELLIA (skip dir)
    no-cast         [option]   OPENSSL_NO_CAST (skip dir)
    no-ec_nistp_64_gcc_128 [default]  OPENSSL_NO_EC_NISTP_64_GCC_128 (skip dir)
    no-gmp          [default]  OPENSSL_NO_GMP (skip dir)
    no-idea         [option]   OPENSSL_NO_IDEA (skip dir)
    no-jpake        [experimental] OPENSSL_NO_JPAKE (skip dir)
    no-krb5         [krb5-flavor not specified] OPENSSL_NO_KRB5
    no-md2          [option]   OPENSSL_NO_MD2 (skip dir)
    no-md5          [option]   OPENSSL_NO_MD5 (skip dir)
    no-mdc2         [option]   OPENSSL_NO_MDC2 (skip dir)
    no-rc2          [option]   OPENSSL_NO_RC2 (skip dir)
    no-rc4          [option]   OPENSSL_NO_RC4 (skip dir)
    no-rc5          [option]   OPENSSL_NO_RC5 (skip dir)
    no-rfc3779      [default]  OPENSSL_NO_RFC3779 (skip dir)
    no-ripemd       [option]   OPENSSL_NO_RIPEMD (skip dir)
    no-seed         [option]   OPENSSL_NO_SEED (skip dir)
    no-srp          [forced]   OPENSSL_NO_SRP (skip dir)
    no-ssl2         [forced]   OPENSSL_NO_SSL2 (skip dir)
    no-ssl3         [forced]   OPENSSL_NO_SSL3 (skip dir)
    no-store        [experimental] OPENSSL_NO_STORE (skip dir)
    no-tls1         [forced]   OPENSSL_NO_TLS1 (skip dir)
    no-tlsext       [forced]   OPENSSL_NO_TLSEXT (skip dir)
    no-zlib         [default] 
    no-zlib-dynamic [default] 
IsMK1MF=0
CC            =gcc
CFLAG         =-DOPENSSL_FIPSCANISTER -O3
EX_LIBS       =
CPUID_OBJ     =mem_clr.o
BN_ASM        =bn_asm.o
DES_ENC       =des_enc.o fcrypt_b.o
AES_ENC       =aes_core.o aes_cbc.o
BF_ENC        =bf_enc.o
CAST_ENC      =c_enc.o
RC4_ENC       =rc4_enc.o rc4_skey.o
RC5_ENC       =rc5_enc.o
MD5_OBJ_ASM   =
SHA1_OBJ_ASM  =
RMD160_OBJ_ASM=
CMLL_ENC      =camellia.o cmll_misc.o cmll_cbc.o
MODES_OBJ     =
ENGINES_OBJ   =
PROCESSOR     =
RANLIB        =/usr/bin/ranlib
ARFLAGS       =
PERL          =/opt/local/bin/perl5
THIRTY_TWO_BIT mode
BN_LLONG mode
RC4_CHUNK is undefined
...