Quite surprisingly, I don't seem to be able to find an answer for this question in Google search.
How many records are too many in an SRV record set?
10, 50 100?
If I want to provide a single hostname that balances across 50 servers in the same physical location, how best to do that?
How do I scale that to 100's of servers?
How many records are too many in an SRV record set?
This all depends on the application consuming them.
Besides the fact that more SRV records mean bigger DNS packets and hence possible problems if broken EDNS support or TCP queries.
This is in fact spelled out by RFC 2782 on SRV:
Currently there's a practical limit of 512 bytes for DNS replies. Until all resolvers can handle larger responses, domain administrators are strongly advised to keep their SRV replies below 512 bytes.
(but it was written on February 2000, so lots of things changed since then).
If I want to provide a single hostname that balances across 50 servers in the same physical location
The DNS might not be the perfect solution for your problem. You can have only one record pointing to a load balancer that will then handle the split between the 50 physical servers.
Note that you might want to look at the newer HTTPS/SVCB records (IETF RFC not yet published, but codepoint allowed already and they are used in the wild and consumed/published by software from Google, Apple, and CloudFlare at least for now).