Set http_proxy, https_proxy on Cloud Composer

Question

We are looking to run a DAG in Cloud Composer that makes https calls to a server outside the network that our Cloud Composer environment lives in. Right now, our calls are timing out and we suspect it is due to our organization's rule of going through a proxy for any calls to external hosts. How do I configure my environment to use the https proxy?

I tried setting an environment variable https_proxy when creating my Composer environment and I got the following error:

    1) Environment variables [http_proxy, https_proxy] may not be overridden.

In short, I would like the virtual machines that are working for my Composer environment to know to use the proxy when making https calls. How can I ensure they use the proxy?

Answer 1

From fellow friend at Google:

"http_proxy and https_proxy variables were recently added to variable blacklist. It is not possible to set these variables anymore, see the release notes here: https://cloud.google.com/composer/docs/release-notes#August_29_2022 . This is due to the reason that it has led to environment stability issues in the past."

My fix:

I set https_proxy inside my task function. I used the python package, os, and added the following line of code at the very top of my task (task that made the https call):

Os.environ[‘https_proxy’] = “{proxy_address}”
Os.environ[‘http_proxy’] = “{proxy_address}”

This seemed to do the trick for me.