I have a site where users can point their own custom domain through the use of DNS records.
For example, someone might point example.com to theirsite.mysite.com so that they are free to use their own domain rather than a subdomain.
Which record would be best for users to set up?
An ALIAS record pointing to mysite.com
An A record pointing to xxx.xx.xx.xx
A CNAME pointing to mysite.com
What are the advantages of using each one?
Which is best depends on the specifics of what you're establishing. Here are the differences:
If you set up an A record, the DNS will resolve to an IP and the end user's browser will make a call to that IP with the host name. That's the call that you have to listen for and handle. Since it's direct to IP right from the start, at scale that IP should be a redirector or load balancer.
Otherwise if you need to switch it to a different machine as an endpoint, you have to deal with inconsistencies on how that traffic is routed due to DNS cacheing and whatever TTL you set expiring. Beyond that the biggest issue you're going to run into is that the customer registers their domain and if it's an A record you give them, they're putting in that IP and to change it you're going to have to get the customer to do it, which is definitely not best practice.
CNAME and ALIAS records are similar. In the brief outline you give above, either of these would seem to be better than an A record. You can give them a domain to enter and that doesn't ever change on their side - but you can switch the end IP they're going to as your architecture expands. There are a few minor differences, but the only significant one is that a CNAME cannot be used as an apex record, e.g. example.com, so most likely you'd need to use an ALIAS record.
If you want to read more about the differences between CNAME and ALIAS, there is a good article on it here.