I'm working on a document management platform. Users can upload documents to the web app, edit their meta-data, and execute tasks (e.g. send the documents to their recipients by e-mail).
Our users need to be able to select any number of documents from a search result. They can pick documents one by one, or click a "Select all" button. We could be talking about thousands or tens of thousands documents to be selected.
When searching for documents, we display paged results in a Datatables table. This means that when making a selection of documents for a task, we need to stock the selection set somewhere, since the Datatable only has knowledge of the rows that are being currently displayed. Currently, we stock the documents' GUIDS in a hidden page element.
This is proving to be insufficient, because we're facing problems with this approach: - When selecting all documents, there might be a concurrency issue. What if more documents are uploaded that fit the current search criteria, between the time a user makes a search and plans a task? This could lead to the unintended processing of documents. - If you send all GUIDs to the client when they choose "Select all" (to avoid the concurrency issue above), the list might be too large to send as a JSON object. - The opposite might also happen: what if the list on the client's side is too large to send to the server?
My question is: what is a better approach to stocking and processing large user selection sets?
I've found out you can compress response data in ASP.Net, making use of GZip or Deflate. Thanks to this, I was able to send large data sets, no problem.