This is what I've cobbled together but the right brace shouldn't contain the underbrace. They should look like siblings, so to speak. Is there a way of doing that?
\(
\left.
\underbrace{
\left[
\begin{matrix}
a_{11} & a_{12} & a_{13} & \cdots & a_{1n} \\
a_{21} & a_{22} & a_{23} & \cdots & a_{2n} \\
a_{31} & a_{32} & a_{33} & \cdots & a_{3n} \\
\vdots & \vdots & \vdots & & \vdots \\
a_{m1} & a_{m2} & a_{m3} & \cdots & a_{mn}
\end{matrix}
\right]
}_{\text{ n columns}}
\right\} \text{m rows}
\)
Here is a complete hack to get this for you.
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/MathJax.js?config=TeX-AMS_HTML"></script>
\(
\kern .4em
\underbrace{
\kern-.4em
\left.
\left[
\begin{matrix}
a_{11} & a_{12} & a_{13} & \cdots & a_{1n} \\
a_{21} & a_{22} & a_{23} & \cdots & a_{2n} \\
a_{31} & a_{32} & a_{33} & \cdots & a_{3n} \\
\vdots & \vdots & \vdots & & \vdots \\
a_{m1} & a_{m2} & a_{m3} & \cdots & a_{mn}
\end{matrix}
\right]
\right\} m\text{ rows}
\Rule{0em}{0em}{3.6em}
\kern -4.5em
}_{\textstyle n \text{ columns}}
\kern 4.5em
\)
The kerning has to be worked out by hand. If you are planning to use this in actual LaTeX, the \Rule
is non-standard, and would have to be replaced by a corresponding \rule
. Or it could be left out entirely (as it is only for a little space between the matrix and the underbrace).