Is it possible to have RStudio render custom LaTeX commands in an rmarkdown document within the editor?
That is, I'd like to be able to see my custom commands rendered on-the-fly, without having to knit the document first, in the same way that standard LaTeX is rendered automatically in the editor view.
Yes, though in my experience support is somewhat spotty.
RStudio uses MathJax for LaTeX display. It's not all of LaTeX, it's a subset designed for displaying math mode expressions in a web page. You can read the details here: https://docs.mathjax.org/en/latest/tex.html.
From the "Defining TeX Macros" section of that web page:
You can use the \def, \newcommand, \renewcommand, \newenvironment, \renewenvironment, and \let commands to create your own macros and environments. Unlike actual TeX, however, in order for MathJax to process such definitions, they must be enclosed in math delimiters (since MathJax only processes macros in math-mode).
So if you have something like
$$
\newcommand{\myexp}{\exp}
$$
in your html_document, you can use \myexp later. But this doesn't work in a pdf_document. As I said, spotty.
Generally speaking I wouldn't use Markdown in RStudio for a serious LaTeX document that needs macro definitions. Use knitr's Sweave-like format in TeXworks or TeXShop, and install a processing engine so knitting is supported.