The best problem description is given by looking at http://rubular.com/r/9zDHFh0gGc. Why is sed not working like that? I think the problem depends on those backslashes \in the fragments, for example in {1+\cos^2 t + t^2}.
In precise I want to substitute in the source
Test Text
= \int_0^{\pi}\vvvec
{1+\cos^2 t + t^2}
{1+ \sin^2 t + t^2}
{1 + 2 t^2}
\cdot \vvvec{-2\cos t \sin t}{2\cos t \sin t}{2t}\ud t =
Ende
by the command sed -r 's/\\vvvec[[:space:]]*\{([^\}]*)\}[[:space:]]*\{([^\}]*)\}[[:space:]]*\{([^\}]*)\}/\\begin\{pmatrix\}\1\\\\\2\\\\\3\\end\{pmatrix\}/g' source and hope to get the result
Test Text
= \int_0^{\pi}\begin{pmatrix}1+\cos^2 t + t^2\\1+ \sin^2 t + t^2\\1 + 2 t^2\end{pmatrix}
\cdot \begin{pmatrix}-2\cos t \sin t\\2\cos t \sin t\\2t\end{pmatrix}\ud t =
Ende
I'm sorry - I can not see, why is is not working.
The string seems to be okay, as echo "\vvvec{1}{2}{3}" | sed -r 's/\\vvvec[[:space:]]*\{([^\}]*)\}[[:space:]]*\{([^\}]*)\}[[:space:]]*\{([^\}]*)\}/\\begin\{pmatrix\}\1\\\\\2\\\\\3\\end\{pmatrix\}/g'
\begin{pmatrix}1\\2\\3\end{pmatrix} is working fine.
It stops working when inserting \sin for example in one component.
The backslash is not needed within a character class, in fact it adds the backslash to the class. Because of this \{([^\}]*)\} won't match {\cos t}.
So, this seems to work:
sed -r 's/\\vvvec[[:space:]]*\{([^}]*)\}[[:space:]]*\{([^}]*)\}[[:space:]]*\{([^}]*)\}/\\begin\{pmatrix\}\1\\\\\2\\\\\3\\end\{pmatrix\}/g'