I am programming a server application (chat server side) with java, which receive requests and send to a target.
Whether caching all or some of the data in memory makes sense depends on your use case. It can improve performance but it adds complexity which might not be needed.
You can load millions or even billions of records into a JVM, but much more then this you need an off heap storage such as a data store for this purpose or a database. Using off heap memory, you can have trillions of records in a JVM but this is rarely needed.