Here's my problem. My GKE GCP node IP addresses have access to an on premise network using ipsec/vpn and on premise firewall rules but my pod IP addresses do not. I want my traffic going from pods to use one of the acceptable node source IP addresses. How can I achieve that?
You should read about IP Masquerarding and how to edit the IP Masquerade agent
IP masquerading is a form of source network address translation (SNAT) used to perform many-to-one IP address translations. GKE can use IP masquerading to change the source IP addresses of packets sent from Pods. When IP masquerading applies to a packet emitted by a Pod, GKE changes the packet's source address from the Pod IP to the underlying node's IP address. Masquerading a packet's source is useful when a recipient is configured to receive packets only from the cluster's node IP addresses.
Please see this post that discusses Egress traffic from GKE Pod through VPN