Not able to Upload large datasets(>10MB) in R Shiny

I am trying to create a tool using R shiny but I am unable to upload large datasets(>10MB) as the load time is increasing and after few minutes the tool crashes. I have also included the option to increase the file size limit of the dataset. Can someone please help me with this query?

The tool is working for smaller datasets which are less than 10 MB.

Sample datasets:

Below are few laptop as well as R version specifications:

' Laptop Specifications: '

Windows 10 System- Processor: 11th Gen Intel(R) Core(TM) i7-11850H @2.50GHz 2.50 GHz Installed Memory (RAM): 16.0 GB(15.7GB usable) System type: 64-bit Operating System, x64-based processor

' R Studio Specification: '

RStudio 2022.07.1+554 "Spotted Wakerobin" Release (7872775ebddc40635780ca1ed238934c3345c5de, 2022-07-22) for Windows Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.8 Chrome/69.0.3497.128 Safari/537.36

app.r

library(shiny)
library(data.table)
library(ff)
library(iotools)
library(readxl)
library(shinycssloaders)
library(shinydashboard)

#UI Code
ui<- shinyUI(fluidPage(
  titlePanel("Demo: Upload Large Dataset > 10 Mb"),
  sidebarLayout(
    sidebarPanel(
      uiOutput("selectfile")
    ),
    mainPanel(
      withSpinner(tableOutput("tb1"),type = 6, color = "#0dc5c1")
      
    )
    
  )
))

#Server Code
server <- shinyServer(function(input, output, session) {
  options(shiny.maxRequestSize=100*1024^2)

dashboard<- read.csv("dataset1.csv")
  
#Display the dataset in the renderTable format 
  output$tb1 <- renderTable({
    dashboard
  })
  
})

#Run App
shinyApp(ui, server)

Solution

With the following small edits the Shiny app works.

Major problem is the renderTable / tableOutput table format, which puts all data on the screen at once, which is never happening with so much data. You can test with smaller data set (like iris) to see. Switching to DT::renderDataTable / DT::dataTableOutput we can have pagination and fast response. https://shiny.rstudio.com/articles/upload.html

Tested moving the loading of data to the top of the application; and this did not matter.

Moved the options command to the top per https://shiny.rstudio.com/articles/upload.html

Switched to fread "fast read" the csv, though that was the not the problem.

library(shiny)
library(data.table)
#library(ff)
#library(iotools)
library(readxl)
#library(shinycssloaders)
#library(shinydashboard)
library(DT)
library(tidyverse)

options(shiny.maxRequestSize=100*1024^2)
#dashboard<- fread("/home/username/Downloads/Airbnb_Open_Data.csv") #testing at the top
#dashboard<-iris  ##A test with only iris data

#UI Code
ui<- shinyUI(fluidPage(
  titlePanel("Demo: Upload Large Dataset > 10 Mb"),
  sidebarLayout(
    sidebarPanel(
      uiOutput("selectfile")
    ),
    mainPanel(
      #withSpinner(tableOutput("tb1"),type = 6, color = "#0dc5c1")
      #tableOutput("tb1")  #simple test without spinner
      DT::dataTableOutput("tb1")
    )
    
  )
))

#Server Code
server <- shinyServer(function(input, output, session) {
  
  
  #dashboard<- read.csv("/home/username/Downloads/Airbnb_Open_Data.csv")
      dashboard<- fread("/home/username/Downloads/Airbnb_Open_Data.csv")

  #Display the dataset in the renderTable format 
#  output$tb1 <- renderTable({
   output$tb1 <- DT::renderDataTable({
      dashboard %>% select(-house_rules)
  })
})

#Run App
shinyApp(ui, server)