Details on the Device:
System: Debian Jessie x86_64 Server (VPS)
Web Server: nginx 1.10.3 (latest stable)
TLS/SSL library: OpenSSL 1.0.2k (just compiled the latest LTS release)
Details on the Question:
Concerns: installing nginx-1.10.3 and nginx-1.11.13 from source (tried both)
Situation: I have a web server up and running already, but nginx is unfortunately compiled with the old OpenSSL 1.0.1t, thus I cannot use HTTP/2 with ALPN yet.
$ nginx -V
nginx version: nginx/1.10.3
built by gcc 4.9.2 (Debian 4.9.2-10)
built with OpenSSL 1.0.1t 3 May 2016
TLS SNI support enabled
So since apt-get install --reinstall nginx won't change that, I figured that I'll have to dpkg-buildpackage or make from the source while both result to the same error.
Problem: Each time I try, the following is the result:
objs/src/event/ngx_event_openssl.o: In function `ngx_ssl_check_host':
/build/nginx-local/nginx-src/nginx-1.10.3/src/event/ngx_event_openssl.c:2997: undefined reference to `X509_check_host'
objs/src/http/ngx_http_request.o: In function `ngx_http_ssl_handshake_handler':
/build/nginx-local/nginx-src/nginx-1.10.3/src/http/ngx_http_request.c:780: undefined reference to `SSL_get0_alpn_selected'
objs/src/http/modules/ngx_http_ssl_module.o: In function `ngx_http_ssl_merge_srv_conf':
/build/nginx-local/nginx-src/nginx-1.10.3/src/http/modules/ngx_http_ssl_module.c:653: undefined reference to `SSL_CTX_set_alpn_select_cb'
collect2: error: ld returned 1 exit status
objs/Makefile:302: recipe for target 'objs/nginx' failed
make[1]: *** [objs/nginx] Error 1
make[1]: Leaving directory '/build/nginx-local/nginx-src/nginx-1.10.3'
Makefile:8: recipe for target 'build' failed
make: *** [build] Error 2
Config: Here is the configuration I used before the make:
$ ./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-file-aio --with-threads --with-ipv6 --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_ssl_module --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
Thanks for the help of the community in advance!
EDIT :
Now I have tried something else. So I modified the sources of nginx-1.10.3, edited the changelog according to the modifications and built the .deb packages using dpkg-buildpackage on on another device (x64 Debian jessie with OpenSSL 1.0.2j) and updated it on my VPS, but this would still result in:
$ nginx -V
nginx version: nginx/1.10.3
built by gcc 4.9.2 (Debian 4.9.2-10)
built with OpenSSL 1.0.1t 3 May 2016
TLS SNI support enabled
The update with the new package has been successful (I changed the sent HTTP Server header to confirm the change and renamed version from 1.10.3-1 to 1.10.3-2)
So the real question is now how I could manage nginx to be built with OpenSSL 1.0.2? I am quite lost at this now.
The modified packages and the source can be found on my website between.
Edit #2:
Even a clean (minimal) install of the VPS without openssl or any server on it was of no help. I mean when compiling OpenSSL 1.0.2k and nginx 1.12 using the original sources (not any debian packages or sources on the repos as Debian usually tends to have old versions anyways) seems to still not work and result in nginx being compiled with openSSL 1.0.1t.
My assumptions:
libssl and libssl-dev (I used apt-get for those packages) could be affecting this?
Somehow nginx still uses the most 'actual' openSSL version available within the Debian repos? (Yeah, Debian still didn't update their openSSL packages since 1.0.1t)
Well, I'll update if I get some more time to try more things out
I solved the same problem by building openssl from source, and addding --with-openssl=../openssl-1.0.2k to nginx's ./configure command.
Maybe an incompatibility with Debian's openssl version ?