How to store a massive number of unique entries?

Question

I am attempting a project which requires me to record each unique URL visited by the user, with the goal being to have a way of storing each array to keep them unique.

Is there an efficient way to do this?

Obviously this could be theoretically massive, possibly tens of thousands of entries if it was an array which would not do - how might I go about storing them?

Answer 1

...possibly tens of thousands of entries if it was an array which would not do...

Arrays can contain tens of thousands, or even hundreds of thousands, of elements.

But for uniqueness, you want a Set rather than an array, as it offers sub-linear lookup time compared to linear lookup time on an array, and better semantics for sets of unique values..

Here's an example building a set of a million unique otherwise-random numbers:

const set = new Set();
console.time("Time to build the set");
while (set.size < 1_000_000) {
    set.add(Math.random()); // Won't add duplicate elements
}
console.timeEnd("Time to build the set");
console.log(`The set contains ${set.size.toLocaleString()} unique otherwise-random numbers`);

For me, that runs in about 200ms. Whereas I gave up on the equivalent using an array after several minutes.