Visual Studio is the recommended tool (superseding Xamarin Studio) on both Windows and Mac. However, the VSTS page Build your Xamarin app says to use "Xamarin". Presumably that means Xamarin Studio, even though the the "Install Xamarin" link on the page points to Visual Studio. That ambiguity and the March 6 date on the page indicate that it was overlooked during the rollout of Visual Studio 7 for Mac, leaving us to wonder what is the right approach for setting up an agent.
What is the best way to create an iOS build agent these days? Do you install Visual Studio for Mac or Xamarin Studio?
Sounds like a bit of confusion here in the terminology. The IDE is now Visual Studio for Mac, but the underlying framework is still Xamarin. That is to say, Xamarin.iOS/Xamarin.Android/Xamarin Forms are "integrated" into Visual Studio for Mac; they are the tools that will compile and package your mobile app. This is what the VSTS build task page is saying when it asks you to "Install Xamarin".
As for setting up a VSTS build agent, here's a checklist that will hopefully get you going:
I'll admit, there is a lot of moving parts and things to do here. I hope this high level overview is enough to point you in the right direction.