I want to distribute the app to only external groups and not the internal groups. But every time I create and upload a build by choosing "Testflight and AppStore", the build compulsorily gets distributed to the internal groups as well, which I don't want to
You can't do this, Internal testers have more access than external testers by default. Internal testers are and should be trusted members.