How to know if it's actually a 404 page?

What I learned from Foregenix:

The HTTP 404 Not Found Error means that the webpage you were trying to reach could not be found on the server. It is a Client-side Error which means that either the page has been removed or moved and the URL was not changed accordingly, or that you typed in the URL incorrectly

But then I also do web app pentests with Python and I am wondering that if I only check for the String 404 on the page, it may not really be a 404 error. It can so happen that the page exists but the heading is 404 just to fool us.

So how exactly do I find out?

Solution

You can check the HTTP status code, and see if it is 404 or not. The status code is on the first line of the response:

HTTP/1.1 404 Not Found

If you are using HTTPlib you can just read the status property of the HTTPResponse object.

However, it is the server that decides what HTTP status code to send. Just because 404 is defined to mean "page not found" does not mean the server can not lie to you. It is quite common to do things like this:

Send 404 instead of 403, to hide the resource that requires authentication.
Send 404 instead of 500, to hide the fact something is not working.
Send 404 when your IP is blocked for some reason.

Without access to the server, it is impossible to know what is really going on behind the curtains.