I am an AWS Architect. I recently walked into an account that had some major design flaws with apps currently getting currently integrated onto the system.
In my opinion, the subnets were just way too large, with a one size fits all mentality that ended up in lazy security groups opening up ports to the entire subnet (/24s and /25s) or even the whole VPC. Also there were routing challenges for some apps being provisioned to the wrong subnets with a long lead time to move them again because of hardened static IP address within the OS. We simply couldn't change the subnet from public to private since other apps resided there. OY!
My Question!
What's your take on subnets and their relationships to security groups when planning for a 50-150 server SaaS (dev,qa, stage, prod) / Corporate Application network (internal apps) environment?
If you have large subnets (example: /24 public, /24 private) for use with all applications, you will need to use security groups attached to your front tier instances as source addresses on security groups further down the stack to be able to both 1) only allow traffic from specific sources on specific service ports 2) be a candidate for autoscaling, right? Otherwise, you would need to manage single IPs as source addresses to restrict traffic which would be 1) a huge pain 2) cant autoscale OR have autoscaling by opening up traffic from the entire subnet/vpc which is not secure.
My alternative subnet network design for smaller subnets was this -
Have smaller subnets (/27s, /28s, /26s) and only allow traffic from corresponding subnets:
Web Tier A (/27) (AZ-A) Web Tier B (/27) (AZ-E)
App Tier A (/27) (AZ-A) App Tier B (/27) (AZ-E)
The App Tier Security Groups only have traffic allowed on the service ports from Web Tiers A & B using the Web Tier A/B CIDRs as source networks . This would allow for both autoscaling and controlled traffic between subnets via security groups. In this model we would not be using other security groups as sources in our SGs, expect perhaps in use with load balancers..
Question for you is.. which one to choose? I feel with the later, smaller, they can be used as building blocks, each app having its own subnetting tier.. slowly carving out the VPC pie. What do you do? What makes sense in terms of operations and scalability?
I'm also an AWS architect, and in my opinion if you're doing access via IP addresses you're doing it wrong.
There are many different ways to separate dev, test, qa and prod environments, and all are valid depending on your requirements. You can completely isolate the different environments by putting them in separate AWS accounts. You can also consider separate VPCs, or simply separate subnets within a VPC. With separate subnets, you can restrict access via routing rules, as well has creating separate sets of security groups for each environment.
Security groups should be granting access to other security groups, not IP addresses. So let's say you have a web app and a database server. Create a "DBAccess" group - it doesn't have to have any rules - and assign it to your web instances. Then create a "DBServer" group, which opens up the appropriate DB port(s) to anything running under the "DBAccess" group. To further restrict access, you'd create DBAccess-Prod, -QA, -Dev, etc. Doing this via cloud formation makes the process simple.
BTW, Amazon has published a NIST 800-53 reference architecture which you might want to look at.