In the GoF design patterns book, when it comes to the implementation part of the Observer pattern, it is stated:
Mapping subjects to their observers The simplest way for a subject to keep track of the observers it should notify is to store references to them explicitly in the subject. However, such storage may be too expensive when there are many subjects and few observers. One solution is to trade space for time by using an associative look-up (e.g., a hash table) to maintain the subject-to-observer mapping. Thus a subject with no observers does not incur storage overhead. On the other hand, this approach increases the cost of accessing the observers.
I fail to see how using hash table would improve storage capacity. In Java, for every subject we could have a list of observers List<Observer>. If there are no observers attached to this subject, the list reference would be null. If we use hash table, Map<Subject, List<Observer>, we still have the list, but we also have a reference to the subject, so this way is a bit more memory
inefficient. I don't know whether it is relevant, but the languages used for implementation in the Gof book are Smalltalk and C++.
The point of the quote seems to be that if subjects are responsible for storing their own observers, in a scenario where most subjects are unobserved at a given time, every subject bears the cost of storing an empty list (imagine millions of subjects).
On the other hand, if subject-to-observer mappings are centralized into a single Map, only the (few) observed subjects have any memory footprint. It is correct to point out that the memory cost per observed subject is higher with a centralized mapping, because of the need to store references to subjects, which is why such a design only makes sense, "when there are many subjects and few observers".
Note a more modern example of optimizing code to avoid empty collections: Why overload the varargs method of() in Java Stream interface?