My R was installed in a conda environment on Ubuntu 20.04.1 VM.
I was searching for this problem when i found this thread. I too am using RStudio from Anaconda, on Debian 10.0 (buster).
Try
par(family = "Arial")
or whichever font name before calling plot(). It worked for me. Update > I still had problems with ggplot2 plots not displaying labels, like above. I tried the workaround of using showtext() instead of regular text rendering.
install.packages("showtext")
After it was installed,
library(showtext)
to load the library. Then simply call
showtext_auto()
followed by the ggplot(.....) + ... + etc. function and text was rendered alright within the plot. I did not load specific fonts using font_add(). Maybe because previously 'Arial' was loaded with par()