I have a website developed using asp.net/C#. I would like to lock an user account on 5 consecutive login failures within a time period of 30 minutes. I do not want to do this on database side. And I know this is cannot be done by session variables. I also do not want to use cookies for this, as a user can easily disable cookies.
Is there a perfect way to do this with above limitations?
Track the number of consecutive failed attempts for a given IP address as well as any given Account ID (username/email). Use a table of failed attempts with columns for IP, date and the account ID. Limit the number of attempted logins over a period of time.
You are correct in stating that you cannot use cookies or session state (which is persisted by a cookie) for this, since an attacker can simply use a fresh cookie or none at all for every attempt, thus fooling your system. Under no circumstances should this be done client-side, as proposed by another answer. The client should never be trusted. But you need to track an attacker somehow, and the only practical way to do that without using cookies, is via their IP address.
IP addresses can be spoofed, but a motivated attacker capable of this en masse is likely to use more sophisticated methods anyhow. You will need to log each login attempt by IP and on every attempt, check if your attempts log contains at least 5 or more attempts within the last N hours for the attempted account. You may also want to limit the total number of attempts for any account from a given IP to prevent an attacker from brute-forcing the same combination on multiple accounts over a period of several hours.
Optionally, you can lock the offending account for several hours (though I am averse to this - your users should never suffer due to your inadequate security) after several consecutive attempts. Bear in mind that any form of successful social engineering and password re-use will thwart your best attempts, so enforcing a strong password policy is paramount.
There are other secondary measures you can take to make attempts statistically not worthwhile, such as automated password resets, with access links sent to the user account. These types of actions would depend on the nature of your product and what you are trying to protect. Banking sites lock accounts with too many consecutive incorrect attempts, for example, because of the severity of a compromised account.
Perhaps the simplest deterrent I would prescribe is to demand a strong CAPTCHA (I recommend ReCAPTCHA) after 3 consecutive failed attempts from any given IP regardless of the attempted username/email. Users behind the same IP may have to enter a CAPTCHA now and then due to other users' failed attempts, but this is a small price to pay for security. As an attacker confronted by ReCAPTCHA, I would simply give up.
A less impinging approach is to limit the number of password attempts by introducing a synthetic authentication delay after 3 consecutive failed attempts. This reduces the viability of a brute-force attack by limiting the number of attempts over N hours which, when combined with an enforced password expiry policy will thwart brute-force attacks.
On a sidenote, make sure you store only salted hashes of user's passwords, with different salts for each password and reject common passwords and dictionary words.