What could be the cause of this error "could not be resolved (110: Operation timed out)"?

Question

I am actually working in a company and to improve SEO, i am trying to setup our angular (10) web app with prerender.io to send rendered html to crawlers visiting our website.

The app is dockerized and exposed using an nginx server. To avoid conflict with existing nginx conf (after few try using it), i (re)started configuration from the .conf file provided in the prerender.io documentation (https://gist.github.com/thoop/8165802) but impossible for me to get any response from the prerender service.

I am always facing: "502: Bad Gateway" (client side) and "could not be resolved (110: Operation timed out)" (server side) when i send a request with Googlebot as User-agent.

After building and running my docker image, the website is correctly exposed on port 80. It is fully accessible when i use a web browser, but the error occurs when i try a request as a bot (using curl -A Googlebot http://localhost:80).

To verify if the prerender service correctly receive my request when needed i tried to use an url generated on pipedream.com, but the request never comes.

I tried using different resolver (8.8.8.8 and 1.1.1.1) but nothing changed.

I tried to increase the resolver_timeout to let more time but still the same error.

I tried to install curl in the container because my image is based on an alpine image, curl was successfully installed but nothing changed.

Here is my nginx conf file :

server {
    listen 80 default_server;
    root /usr/share/nginx/html;
 
    index  index.html;

    location / {
        try_files $uri @prerender;
    }
 
    location @prerender {
        proxy_set_header X-Prerender-Token TOKEN_HERE;
        
        set $prerender 0;
        if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp") {
            set $prerender 1;
        }
        if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
        }
        if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
        }
        if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
            set $prerender 0;
        }
        
        #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
        resolver 8.8.8.8;
        resolver_timeout 60s;
 
        if ($prerender = 1) {
            
            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "service.prerender.io";
            rewrite .* /$scheme://$host$request_uri? break;
            proxy_pass http://$prerender;
        }
        if ($prerender = 0) {
            rewrite .* /index.html break;
        }
    }
}

And here is my Dockerfile:

FROM node:12.7-alpine AS build
ARG environment=production

WORKDIR /usr/src/app
COPY package.json package-lock.json ./
RUN npm install
COPY . .
RUN npm run build -- --configuration $environment
# Two stage build because we do not need node-related things
FROM nginx:1.17.1-alpine

RUN apk add --no-cache curl
COPY --from=build /usr/src/app/dist/app /usr/share/nginx/html
COPY prerender-nginx.conf /etc/nginx/conf.d/default.conf

EXPOSE 80

Hope you will have a track or an idea to help me

Answer 1

Erroneous part would be

curl -A Googlebot http://localhost:80

The way how prerender works, is accesses the fqdn you have sent to original webserver. So localhost:80 will not be accessible.

Try passing proper hostname, kind of

curl -H "Host: accessiblefrom.public.websitefqdn:80"  http://localhost:80

Check out example on https://github.com/Voronenko/self-hosted-prerender