We have an azure VPN gateway setup with two site-to-site connections that are successfully configured. Site A (local network gateway A) and Site B (local network gateway B) can each see everything in the azure VNET where the VPN gateway resides.
We cannot, however, see resources between our sites. For example, I cannot RDP to a machine from my local network gateway A to a machine in local network gateway B even though both local network gateways are connected to the Azure VPN gateway. Both sites (A and B) have added static routes to ensure our respective routers forward requests through the VPN tunnel for our respective local gateway address spaces.
Is this type of connectivity turned off by default for security reasons or should it work with default settings?
This is possible only with a Route Based VPN, and will work best with BGP Enabled. You might also need to create custom routes on your on-prem devices so that traffic destined for your other on-prem network will go to the azure gateway.