So far I was able to redirect TCP connections with a specific destination address or port to my own program with this iptables rule:
iptables -t nat -A OUTPUT -p tcp -d <address> --dport <port> -j REDIRECT --to <local_port>
This works well until I create a connection to this destination from my proxy because it recursively connects to itself.
Is there a way for iptables to know what the original connection is and only redirect it? Or is there a better approach?
You can try using owner module and skip the redirection for the traffic coming from the proxy. Check for --uid-owner or --pid-owner, you should be able to differentiate the traffic based on either of these.
Something like this,
iptables -t nat -I OUTPUT -m owner -p tcp -d <address> --dport <port> --uid-owner <proxy-owner> -j ACCEPT