One interesting question that was thrown to me was whether the re-arranging of the memory address from tag, cache index, offset to cache index, tag, offset would yield any difference? Is there any significant issues that may arise if the most significant bit of the memory address is now the cache index instead of the tag? How those the cpu access the bits? Sequentially or all at once?
Three factors make indexing a cache by more significant bits unattractive. First, using only more significant bits for indexing would tend to increase conflict misses. Spatial locality is common at granularity greater than the cache block size (otherwise modest sized TLBs would not be effective). For example, any stack frame using more than one cache block would have conflicts, using up associativity, and adjacent frames would likewise conflict. (TLBs would be much less effective if spatial locality was limited to within a cache block. This would also prevent using "pre-validated" tags — where the TLB entry number is used for a tag — as several Itanium implementations did; "Don't use the page number, but a pointer to it" (André Seznec, 1996) describes the basic concept.)
Second, for virtual address systems that allow synonyms (a physical page mapping to two different virtual addresses), one could not use the straightforward technique of using only bits within the page offset to map all synonyms into the same set. Other techniques are available to handle synonyms, but this method is commonly used.
Third, by indexing the cache using less significant bits, the cache access can be started before address calculation has completed. Since base+offset is a common addressing mode, such is typically optimized for latency. Addition proceeds from the least significant bits to the most significant bits, so the least significant bits of the result are available earlier.