I am trying to get some feedback from people using AWS regarding whether or not we can have a rough estimate from an HPC cluster configured on AWS as following
[aws]
aws_region_name = us-east-2
[cluster default]
key_name = xxxx
vpc_settings = vpc-xxxxx
compute_instance_type = c4.8xlarge
master_instance_type = c4.large
initial_queue_size = 4
max_queue_size = 20
maintain_initial_size = true
scheduler = sge
cluster_type = spot
spot_price = 1.60
placement_group = DYNAMIC
master_root_volume_size = 20
base_os = ubuntu1404
ebs_settings = custom
[ebs custom]
volume_type = io1
volume_iops = 2000
volume_size = 2000
encrypted = true
[vpc vpc-xxx]
vpc_id = vpc-xxx
master_subnet_id = subnet-xxxx
[global]
cluster_template = default
update_check = true
sanity_check = true
[aliases]
ssh = ssh {CFN_USER}@{MASTER_IP} {ARGS}
Based on how long jobs will be running on the cluster, do you think it is possible to sort of roughly have an idea on how much it is going to cost on AWS ? A lot of things are involved in there, EC2 costs, storage costs etc
Given these will be spot instances (compute) rather than On demand, how can we get an estimation
Thanks for your ideas
First of all, I have good news for you. The price for on-demand c4.8xlarge instances is currently $1.591 per Hour in your region. This is also the highest possible price you can pay for a c4.8xlarge spot instance. Further, it is the CfnCluster default to use the current spot market price, so just remove this line from your config file.
As to your actual question: It is complicated.
You can of course easily calculate a maximum price. Your compute instance will cost you a maximum of 20*1.591=31.82$ per hour. However, the actual price will be much less. By using spot instances, you can easily expect the price to be half of what you would pay for on-demand instances. Also, your cluster will probably not always run at full capacity.
You can check the current spot prices at https://aws.amazon.com/ec2/spot/pricing/.
Your shared EBS volume will cost around 1000*0.125=125$ per month as can be seen at https://aws.amazon.com/ebs/pricing/
To summarize, you should probably assume to pay around 20 to 40% of the on-demand prices for your spot instances, depending on how conservative you want to estimate your costs. Based on this, you can estimate how much you will pay for a specific amount of cluster usage hours. Of course, you also need to estimate how many nodes will be running on average. With your current configuration, this will vary between 4 and 20.