I've read couple of articles about AWS Security Groups:
but couldn't find at which OSI layer Security Groups operate. Maybe someone can help me understand at which OSI layer they operate?
Security Groups make it possible to create traffic allow rules based on IPs, protocols and optionally ports. IPs are associated with Layer 3 (Network) of the OSI-model and Ports with Layer 4 (Transport). The protocols you can filter on are a little tougher - you can select some from layer 3 (e.g. ICMP), layer 4 (e.g. TCP/UDP) or layer 5+ (HTTP,...) in the console. If you select one of the layer 5+ protocols, it will actually set TCP or UDP for you.
As a result of that, I would say that the security group operates at both layer 3 and layer 4 of the OSI model. I'm not sure if that really fits in the model, but I'm not very dogmatic about the model - as the statisticians say: All models are wrong, but some are useful.