Loading small amounts of data from Lumen API time

Question

I have an API built in lumen and I plan to consume the API json response in the frontend using a single page framework like Angular.

The problem is the response from some routes contain huge amount of data which is aprox 50000 rows as of now. I intend to send the data to the frontend in small portion depending on how much the user is using.

for example on a social site you get small chunks of posts loaded at a time and as you scroll down more post are loaded until you eventual close the application

I would like my API to do something similar. Alternatives that I have considered are

1. Using the Laravel Chunk method. The problem with this method I have found that chunking does not necessarily bring small amounts of data at time, it just sub divides it and I can't use it because the server will load for a very long time if the data to be subdivided is huge as in my case and will eventually run out of memory and it is bad UX.
2. Using the Laravel Paginate method. Even though this does exactly what I described above, the issue I have is that I want the data to be in list format and lazyloaded whenever a user reaches the end of a page exactly the way facebook does when you reach the last post. I don't want users to paginate to view more data. I would like the API to keep sending data depending on where it left.

What options do I have for achieving this and also is there a workaround on the options I mentioned above?

THANK YOU

Answer 1

So, since the Paginate method does exactly what you need, you will need to use it. Implement an API function that will receive a starting point and an offset on the server-side.

On the client-side you do not have to display paging controls. You can instead implement a scroll event and detect if the scrolling reached the bottom. If so, then trigger a function (still in JS on the client-side), that will ask for the next packet of data. For this purpose, you will always need to know/find out how many records were already loaded and send that number as the starting point and your offset to the API function we have discussed above.