Search code examples
javascriptmongodbexpressoptimizationweb-applications

What should I do when my database is too huge and I need all the data initially at the frontend


I use Mongodb and my collection has gotten very huge. It contains 15k documents and it's size is almost 15 mb. The website I am working on uses a map and shows every element on the map. This design of the website makes it hard to paginate. Whenever I make a request to my express backend, it gives me error about the javascript heap out of memory.This is that error

What can I do to make sure my website still works while keeping these things in mind:

  1. My database is too huge
  2. My website doesn't allow pagination
  3. If I keep using the current design my backend crashes

I'm sorry if I am being stupid or something, this is my first time asking a question on stackoverflow


Solution

  • Don't do pagination, but still load the data incrementally, and visualize it as it arrives. Think of it as a loading animation.

    Step 1: Load app, display the map without data Step 2: As the data arrives in chunks render it.

    Or use something like websocket.

    Find a way to hash/paginate it, even if you use something stupid like first character of the ID to do it