Search code examples
javascriptphpajaxperformancelarge-data

Best way to Handle large amounts of Data

Currently I am experiencing speed issues with parts of my application that load large amounts of data into a reporting table. The data in the reporting table is pulling from multiple tables and running some complex queries, but required quires.

Besides optimizing the code, my question is how do you personally handle large amounts of data that need to be displayed to the user, and what is the best practice?

Currently I am processing all the data before hand and then generating a table via the data table javascript library.

Things I know:

  • The user doesn't need to see all the data at once
  • The user needs to be able to search through all the data
  • The user needs to be able to filter through the data

Is the best way really to just use a loading spinner, and load only a small portion of the data when the page first loads? then the rest of the data retrieval is through Ajax?

I feel like there has to be a better way

Thanks,

Solution

  • I think you're answering your own question a bit. Yes, it is better to not deliver the whole database to the user at once, this is why any RDBMS supports things like LIMIT. Your three criteria exactly match what a database system can do for you – queries of small subsets of data (i.e. pages), optionally filtered or matched based on a search query.

    For simplicity of the front end, you can make the first page load via AJAX as well, though having it pre-rendered does make the page feel more responsive. Having said that, there are many existing solutions to this problem; some template engines and JS front-end frameworks (Vue.js SSR) support server-side pre-render.