I am currently setting up a High Availability (HA) environment with two Azure Virtual Machines running Ubuntu sitting behind a standard Azure load balancer. Now I know the standard load balancer is only layer 4 which means it cannot do SSL offloading.
The two VM's are both running a .NET Core Web API. They would obviously each need the SSL certificate to handle SSL connections coming in from the load balancer.
I know I can purchase an SSL certificate and just setup Kestrel to use the certificate on the Web API itself but I would like the free certificate. I know the other option is to generate the certificate using an nginx server then copy across the certificates to the Web API but this means I would need to repeat the process every 3 months which is quite a ball ache as it means I would have downtime while I take the HA cluster offline to renew the certificate.
Does anyone know of a way to use Lets Encrypt on the two VMs sitting behind the load balancer?
Okay so I came right with the above. It required me to write a utility that auto renews my Lets Encrypt certificates using DNS verification. It is quite important that it uses Azure DNS or another DNS provider that has an API as you will need to be able to modify your DNS records directly with either an API or some other interface with your provider.
I am using Azure DNS and it is managing the entire domain for me so the code below is for Azure DNS but you can modify the API to work with any provider of your choosing that has some sort of API.
The second part of this, is not to have any downtime in my high availability (HA) cluster. So what I have done is, is to write the certificate to the database and then read it dynamically on startup of my VM's. So basically every time Kestrel starts it reads the certificate from the DB and then uses that.
You will need to add the following model to your database so that you can store the actual certificate particulars somewhere.
public class Certificate
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public long Id { get; set; }
public string FullChainPem { get; set; }
public string CertificatePfx { get; set; }
public string CertificatePassword { get; set; }
public DateTime CertificateExpiry { get; set; }
public DateTime? CreatedAt { get; set; }
public DateTime? UpdatedAt { get; set; }
}
Once you have created the model you will need to place it in your context as follows:
public DbSet<Certificate> Certificates { get; set; }
On your application servers you would want to use Kestrel to act as the web server and then load the certificate dynamically from the database. So add the following to your CreateWebHostBuilder method. It is important that this is after .UseStartup<Startup>()
.UseKestrel(opt = >{
//Get the application services
var applicationServices = opt.ApplicationServices;
//Create and use scope
using(var scope = applicationServices.CreateScope()) {
//Get the database context to work with
var context = scope.ServiceProvider.GetService < DBContext > ();
//Get the certificate
var certificate = context.Certificates.Last();
var pfxBytes = Convert.FromBase64String(certificate.CertificatePfx);
var pfxPassword = certificate.CertificatePassword;
//Create the certificate
var cert = new X509Certificate2(pfxBytes, pfxPassword);
//Listen on the specified IP and port
opt.Listen(IPAddress.Any, 443, listenOpts = >{
//Use HTTPS
listenOpts.UseHttps(cert);
});
}
});
So this is the meat of the solution. It handles the certificate requests, challenges, DNS verification and then the storage of the certificates. It also will auto restart each VM instance in Azure that is using the certificates so that they pull the new certificates.
The Main logic is as follows, it will check whether the certificates need to be renewed or not.
static void Main(string[] args) {
while (true) {
//Get the latest certificate in the DB for the servers
var lastCertificate = _db.Certificates.LastOrDefault();
//Check if the expiry date of last certificate is more than a month away
if (lastCertificate != null && (lastCertificate.CertificateExpiry - DateTime.Now).TotalDays > 31) {
//Log out some info
Console.WriteLine($ "[{DateTime.Now}] - Certificate still valid, sleeping for a day.");
//Sleep the thread
Thread.Sleep(TimeSpan.FromDays(1));
}
else {
//Renew the certificates
RenewCertificates();
}
}
}
Okay so this is a lot to go through but it is actually quite simple if you break it down
The actual RenewCertificates method is as follows:
/// <summary>
/// Method that will renew the domain certificates and update the database with them
/// </summary>
public static void RenewCertificates() {
Console.WriteLine($ "[{DateTime.Now}] - Starting certificate renewal.");
//Instantiate variables
AcmeContext acme;
IAccountContext account;
//Try and get the setting value for ACME Key
var acmeKey = _db.Settings.FirstOrDefault(s = >s.Key == "ACME");
//Check if acme key is null
if (acmeKey == null) {
//Set the ACME servers to use
#if DEBUG
acme = new AcmeContext(WellKnownServers.LetsEncryptStagingV2);
#else
acme = new AcmeContext(WellKnownServers.LetsEncryptV2);
#endif
//Create the new account
account = acme.NewAccount("yourname@yourdomain.tld", true).Result;
//Save the key to the DB to be used
_db.Settings.Add(new Setting {
Key = "ACME",
Value = acme.AccountKey.ToPem()
});
//Save DB changes
_db.SaveChanges();
}
else {
//Get the account key from PEM
var accountKey = KeyFactory.FromPem(acmeKey.Value);
//Set the ACME servers to use
#if DEBUG
acme = new AcmeContext(WellKnownServers.LetsEncryptStagingV2, accountKey);
#else
acme = new AcmeContext(WellKnownServers.LetsEncryptV2, accountKey);
#endif
//Get the actual account
account = acme.Account().Result;
}
//Create an order for wildcard domain and normal domain
var order = acme.NewOrder(new[] {
"*.yourdomain.tld",
"yourdomain.tld"
}).Result;
//Generate the challenges for the domains
var authorizations = order.Authorizations().Result;
//Error flag
var hasFailed = false;
foreach(var authorization in authorizations) {
//Get the DNS challenge for the authorization
var dnsChallenge = authorization.Dns().Result;
//Get the DNS TXT
var dnsTxt = acme.AccountKey.DnsTxt(dnsChallenge.Token);
Console.WriteLine($ "[{DateTime.Now}] - Received DNS challenge data.");
//Set the DNS record
Azure.SetAcmeTxtRecord(dnsTxt);
Console.WriteLine($ "[{DateTime.Now}] - Updated DNS challenge data.");
Console.WriteLine($ "[{DateTime.Now}] - Waiting 1 minute before checking status.");
dnsChallenge.Validate();
//Wait 1 minute
Thread.Sleep(TimeSpan.FromMinutes(1));
//Check the DNS challenge
var valid = dnsChallenge.Validate().Result;
//If the verification fails set failed flag
if (valid.Status != ChallengeStatus.Valid) hasFailed = true;
}
//Check whether challenges failed
if (hasFailed) {
Console.WriteLine($ "[{DateTime.Now}] - DNS challenge(s) failed, retrying.");
//Recurse
RenewCertificates();
return;
}
else {
Console.WriteLine($ "[{DateTime.Now}] - DNS challenge(s) successful.");
//Generate a private key
var privateKey = KeyFactory.NewKey(KeyAlgorithm.ES256);
//Generate certificate
var cert = order.Generate(new CsrInfo {
CountryName = "ZA",
State = "Gauteng",
Locality = "Pretoria",
Organization = "Your Organization",
OrganizationUnit = "Production",
},
privateKey).Result;
Console.WriteLine($ "[{DateTime.Now}] - Certificate generated successfully.");
//Get the full chain
var fullChain = cert.ToPem();
//Generate password
var pass = Guid.NewGuid().ToString();
//Export the pfx
var pfxBuilder = cert.ToPfx(privateKey);
var pfx = pfxBuilder.Build("yourdomain.tld", pass);
//Create database entry
_db.Certificates.Add(new Certificate {
FullChainPem = fullChain,
CertificatePfx = Convert.ToBase64String(pfx),
CertificatePassword = pass,
CertificateExpiry = DateTime.Now.AddMonths(2)
});
//Save changes
_db.SaveChanges();
Console.WriteLine($ "[{DateTime.Now}] - Database updated with new certificate.");
Console.WriteLine($ "[{DateTime.Now}] - Restarting VMs.");
//Restart the VMS
Azure.RestartAllVms();
}
}
Wherever I called Azure you would need to write your API wrapper to set DNS TXT records, and then the ability to restart the VMs from your hosting provider. Mine was all with Azure so it was pretty simple to do. Here is the Azure code:
/// <summary>
/// Method that will set the TXT record value of the ACME challenge
/// </summary>
/// <param name="txtValue">Value for the TXT record</param>
/// <returns>Whether call was successful or not</returns>
public static bool SetAcmeTxtRecord(string txtValue) {
//Set the zone endpoint
const string url = "https://management.azure.com/subscriptions/{subId}/resourceGroups/{resourceGroup}/providers/Microsoft.Network/dnsZones/{dnsZone}/txt/_acme-challenge?api-version=2018-03-01-preview";
//Authenticate API
AuthenticateApi();
//Build up the body to put
var body = $ "{{\"properties\": {{\"metadata\": {{}},\"TTL\": 225,\"TXTRecords\": [{{\"value\": [\"{txtValue}\"]}}]}}}}";
//Build up the string content
var content = new StringContent(body, Encoding.UTF8, "application/json");
//Create the response
var response = client.PutAsync(url, content).Result;
//Return the response
return response.IsSuccessStatusCode;
}
I hope this is able to help someone else that was in the same predicament as myself.