Handling large data sets on client side

Question

I'm trying to build an application that uses Server Sent Events in order to fetch and show some tweets (latest 50- 100 tweets) on UI.

Url for SSE:

https://tweet-service.herokuapp.com/stream

Problem(s):

• My UI is becoming unresponsive because there is a huge data that's coming in!
• How do I make sure My UI is responsive? What strategies should I usually adopt in making sure I'm handling the data?

Current Setup: (For better clarity on what I'm trying to achieve)

• Currently I have a Max-Heap that has a custom comparator to show latest 50 tweets.
• Everytime there's a change, I am re-rendering the page with new max-heap data.

Answer 1

We should not keep the EventSource open, since this will block the main thread if too many messages are sent in a short amount of time. Instead, we only should keep the event source open for as long as it takes to get 50-100 tweets. For example:

function getLatestTweets(limit) {
  return new Promise((resolve, reject) => {
    let items = [];
    let source = new EventSource('https://tweet-service.herokuapp.com/stream');

    source.onmessage = ({data}) => {
      if (limit-- > 0) {
        items.push(JSON.parse(data));
      } else {
        // resolve this promise once we have reached the specified limit
        resolve(items);
        source.close();
      }
    }
  });
}

getLatestTweets(100).then(e => console.log(e))

You can then compare these tweets to previously fetched tweets to figure out which ones are new, and then update the UI accordingly. You can use setInterval to call this function periodically to fetch the latest tweets.