I am using Google Cloud Composer to schedule workflows by running some scripts with dependencies.
These scripts (bash and python scripts) are in a virtual machine hosted on Oracle Cloud Infrastructure in a private network. The scripts connect to another SFTP location to retrieve files and then process them.
How do I go about connecting to the VM inside a private network which has those scripts and executing them from Cloud Composer.
I was thinking maybe a SSHExecuteOperator for the bash script but for the python script, PythonOperator? But how do I connect to the private network, using a vpn capability or creating a connection in the UI and use SSH without password?
You have to use the SSH operator:
The PythonOperator is to execute local Python code, not remotely.
About the private connectivity, you have to connect you Composer VPC to the external network. You can use VPN or Interconnect according with the level of reliability that you need and the bandwidth required.
When the connection is set up, think to open correctly the firewall rules and the routes (in the 2 ways! I often forget to open the route back!). Then you simply have to reach the private IP of your VM to connect it (be careful, you might have to translate it in case of IP range overlapping).