Is it possible to create a DNS subdomain containing special characters?
For example, is *.example.com or $.example.com valid according to the RFC for DNS?
The short answer to your question boils down to "Yes, but no, but sometimes yes".
At the protocol level, DNS strings (including names) are encoded as length+data, so the data can be anything. So in that way * and $ are perfectly fine.
The level above the protocol is the human-name level. On that level there are restrictions on what names you can use. Since the 80s, the main restriction boils down to letters, numbers and - (as long as it's not at the beginning or end of a label). So in that way * and $ are forbidden (except that * as the entire content of a label has a special meaning).
On top of that, these days we have internationalized names. That's a way to encode any Unicode string into a form that conforms to the above rule. This, way we can have names that look like räksmörgås.se to humans while they internally look like xn--rksmrgs-5wao1o.se. That xn-- at the start is a prefix that says that this is an encoded name. You still can't use * or $ in your names, but you can probably find something else in Unicode that looks close enough and that you can use... which is a security problem of its own.
The specification for all this is spread out over far too many RFCs. If you're curious, start here and follow many, many links from there.