I can set them up one-by-one with self-signed certs, but it's not practical for how many subdomains I'm working with.
Is there an easier way to do this?
Easiest is to use a wildcard certificate signed by a recognized authority. That also makes the certificates valid and validatable which is not the case for self signed certificates.
A wildcard certificate costs a fee for the signing service. Some 20 Euros a year currently. And you obviously have to go through a verification process, typically per telephone. The Let's Encrypt certificates are great, but they do not offer wildcard certificates. You certainly can automate the creation of their certificates by scripting, though, if you are willing to invest effort into that.
I personally used startssl in the past. Their portal is a bit difficult to use, but things work, and I failed to find anything comparable if it comes to prices. You need their "level 2" for a wildcard certificate, the free certificates are always for a single host name only. Each wildcard certificate will cost extra, obviously, but similar obviously you can use a single wildcard certificate with all host names within a given domain name which is what you are looking for I think.