I didn't save the notebook but as in rstudio these files are still there with names like Untitled.Rmd and Untitled.nb.html. I understand the problem is that the file has a lot in it (tons of intermediate values in parameter optimization) and I noticed before the studio has problem loading when there is a lot of stuff in notebook. When I had this problem with saved notebook, I simply deleted them and the problem was solved. Now the problem is I don't know where the Untitled.Rmd and Untitled.nb.html are saved.
I found the location where these files are saved. It's in the following folder:
~/.local/share/rstudio/sources
There is bunch of folders starting with s- . The unsaved .Rmd file which was giving me problem was saved in folder named s-6ABEC66A.