Search code examples
azureazure-application-gateway

Setting Up Prometheus, Grafana, and Loki on Azure VM with HTTPS Using Azure Application Gateway

Question:

Hi everyone,

I'm fairly new to using Azure Application Gateway and need some assistance with a setup involving Prometheus, Grafana, and Loki on a single Azure VM. I want to expose these services securely over HTTPS through distinct subdomains (e.g., prometheus.example.com, grafana.example.com, loki.example.com) and manage DNS with AWS Route 53.

My Setup:

  1. Azure VM: Hosting Prometheus, Grafana, and Loki.

  2. Azure Application Gateway: To route traffic based on subdomains.

  3. HTTPS: Enforce HTTPS only for all services.

  4. DNS: Managed by AWS Route 53.

Objectives:

  • Prometheus: Accessible via https://prometheus.example.com on port 9090.

  • Grafana: Accessible via https://grafana.example.com on port 3000.

  • Loki: Accessible via https://loki.example.com on port 3100.

  • Scalability: Plan to add more applications using the same Azure Application Gateway.

Current Progress:

  1. VM Setup: Created an Azure VM and installed Prometheus, Grafana, and Loki.

  2. Networking: Configured Azure VNet and NSG to manage network traffic.

  3. Application Gateway: Set up with listeners and backend pools.

  4. DNS: Managed via AWS Route 53 to point subdomains to the Application Gateway's public IP.

Challenges I'm Facing:

  1. Routing Traffic: Need guidance on configuring Azure Application Gateway to correctly route traffic to Prometheus, Grafana, and Loki based on subdomains.

  2. HTTPS Enforcement: How to ensure HTTP requests are redirected to HTTPS for all subdomains.

  3. Health Probes: Best practices for setting up health probes for these services.

  4. SSL Certificate Management: Does Azure have a certificate manager like AWS Certificate Manager? How can I manage SSL certificates in Azure, without manually created certificates or using Let's Encrypt?

Here’s my current configuration:

  • Application Gateway Listeners:

    • Listener for prometheus.example.com on port 443

    • Listener for grafana.example.com on port 443

    • Listener for loki.example.com on port 443

  • Backend Pools:

    • VM with Prometheus, Grafana, and Loki installed

  • HTTP Settings:

    • Prometheus: http://<VM private IP>:9090

    • Grafana: http://<VM private IP>:3000

    • Loki: http://<VM private IP>:3100

  • DNS Records in AWS Route 53:

    • A records pointing subdomains to the Azure Application Gateway’s public IP.

Specific Questions:

  1. Routing Configuration: How do I configure Azure Application Gateway to route traffic correctly to Prometheus, Grafana, and Loki based on subdomains?

  2. Health Probes: What are the best practices for setting up health probes for these services?

  3. HTTPS Redirection: How can I ensure that HTTP requests are redirected to HTTPS for all subdomains?

  4. SSL Certificate Management: Does Azure have a built-in certificate manager similar to AWS Certificate Manager? If so, how can I use it to manage my SSL certificates? Alternatively, how can I manage SSL certificates without manually created certificates or using Let's Encrypt?

Additional Context:

I’m excited about leveraging Azure Application Gateway for this setup, as it will allow me to add more applications easily in the future. Any guidance, step-by-step instructions, or examples would be greatly appreciated!

Thank you in advance for your help!

Solution

  • ...configuring Azure Application Gateway to correctly route traffic to Prometheus, Grafana, and Loki based on subdomains.

    Assuming you have configured the listeners for the frontend for each application (differentiated by port), you should also have different backend pools corresponding to each frontend.

    1. Configure the HTTP settings for the backends. Ensure the protocol matches that which would be used if you were browsing directly to the VM. Ie. Plain HTTP, in most cases. If you want to use HTTPS, you will need to ensure you configure each of your applications accordingly, and either use a certificate issued by a public CA, or import the CA cert in to App Gateway in Trusted CAs.
    2. Create URL path maps in App Gateway. You will need a single path map for each application. The path should be /, and you must specify in the condition that the host header must match the domain name for that particular application.
    3. Set up routing rules that map your HTTPS frontend listener to the path maps you created. One rule per backend application.

    What are the best practices for setting up health probes for these services?

    You need to choose a strategy, either from the product documentation or by devising one yourself. Usually, a probe configured to do GET / is sufficient to verify that the host is accessible, and that the web server is responding fine. If the root redirects with 301 or 302, use the path it redirects to instead. For example, /login.php. You should be probing for a 20x HTTP response code.

    How can I ensure that HTTP requests are redirected to HTTPS for all subdomains?

    1. Create a frontend listener for plain HTTP, port 80. Leave the hostname blank to apply to all plain HTTP requests coming in.
    2. In App Gateway -> Settings -> Redirect Configurations, add a configuration. Select the target as your HTTPS listener. Select the option to include query string. Decide whether you want a permanent redirect 301, which is cached by upstream proxies and browsers, or a temporary 302 which just means that the client will make a request every time to check that a redirect is still in place. 301 permanent should be fine if you don't ever intend on serving HTTP on this gateway.

    Does Azure have a built-in certificate manager similar to AWS Certificate Manager? If so, how can I use it to manage my SSL certificates? Alternatively, how can I manage SSL certificates without manually created certificates or using Let's Encrypt?

    Not quite. There is Azure Managed Certificates for App Services, and Azure KeyVault supports generating private keys and CSRs, but you need a public CA to sign your certificate, such as GlobalSign, Digicert etc. If you already have a wildcard cert for *.example.com you can use this. Or, indeed you could use Let's Encrypt with DNS01 verification. Ultimately, you can upload your signed PFX format certificate chain (must include intermediates and root CA) and private key directly to App Gateway, or reference an existing certificate in KeyVault (which you would need to create or upload). I won't cover the KeyVault steps here because it's bordering going off topic.

    The upshot is, there isn't currently any out-of-box functionality for certificate issuance or renewal. You could however use a cert issues in AWS Certificate Manager.