If I have a large scale system, which normal form I should use: 3NF, BCNF, 4NF, 5NF or higher?
It depends. What are you going to be doing with the data?
Is your database intended to support on line transaction processing (OLTP)? Or is it intended to support on line analytic processing (OLAP), or reporting, data mart, or data warehouse activity?
In the OLAP case, you might want to consider designing a star or snowflake schema and not worry about normal forms. In the OLTP case, a normalized database might give you better results than a less normalized one.
How important is correct data? A database that contradicts itself can be a real mess. Two outputs that are supposed to be equivalent contradict each other instead? How can this happen?
Well, if a database stores the same fact in several places in the database, it might somehow manage to store different and mutually contradictory versions of that fact in diferent places. How would a database store multiple copies of a fact in several places? If it's not completely normalized.
Associated with each normal form is one or more update anomalies that can occur when rows are inserted, updated, or deleted. These anomalies are obviated by the corresponding normal form. You can avoid this problem by careful programming in your updates, but obviating is more foolproof than avoiding. Review the normal forms, if necessary, to familiarize yourself with the anomalies, and decide how big a problem they are in your case.
How important is performance at update time? at query time?
Some people urge you to normalize in order to save space in the database. Disk space is cheap. Some people worry about processing time. This is generally trivial. Delays due to extra disk accesses are noticeable, but often manageable.
There is a place where failure to normalize can result in a performance disaster, however. It's bottlenecks associated with heavy loads and conservative concurrency control. Most DBMS servers adopt a conservative concurrency control policy in order to protect the data from mysterious timing dependent bugs like phantom updates. Even if you can loosen the concurrency control policy, you do so at your peril.
A poorly normalized database often has these bottlenecks or "hot spots". They won't surface when the system is under a light load. The system may make it through beta testing with flying colors only to slow down to a crawl in actual production. Databases that back up websites are notorious for having this flaw. Normalizing can help you avoid this situation by keeping update transaction simple.
So what should you aim for? When I was building databases, I generally aimed for 3NF or BCNF. 3NF is really easy. You just make sure that non key data depends on the key, the whole key, and nothing but the key (so help me Codd). I generally never had to worry about 4NF or 5NF, but again, it depends on your case.