AWS : How to guarantee availability of static Private IP while bootstrapping

I am launching EMR cluster and while bootstrapping I am assigning a Private IP address to it master node. So this get associated when cluster starts and gets released when cluster gets terminated. How do i guarantee that this IP is always available for my cluster and no other process picks it up even when not in use.

Once I get this Ip, I submit my spark jars to 8998 port of this IP, using Apache livy REST API

So my use case is to expose the IP on master node, so that LIVY APIs can submit job to EMR.

Solution

It is not possible to request a specific private IP address for the cluster.

Instead, I would suggest:

Create a Route 53 private hosted zone with a Record Set for the cluster (eg cluster.private)
After launching the EMR cluster, update the Record Set for the IP address of the master node
Have Apache Livy reference the cluster by DNS Name rather than IP address

Updating the Record Set can be done programmatically. A script would do something like:

Call list_clusters()
For each cluster, call describe_cluster()
Look for a tag (eg DNS=cluster.private)
If the DNS tag is present:
- Update the named Record Set in Route 53 to use an A-Record to point to the IP address of the master node

This script would need to be triggered after the cluster is launched.