How do I migrate .NET Core data protection keys from Redis to MySQL entity framework?

Question

I currently have my data protection keys stored in Redis, but I would like to migrate them into a database running MySQL.

How can I migrate my existing data protection keys from Redis to MySQL?

Info about data protection keys: https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/implementation/key-storage-providers?view=aspnetcore-6.0

Answer 1

I have just done this, so here's a write-up of how I did it. Hopefully someone will find it useful.

Pre-requisites:

• Ensure you can connect to your Redis instance via redis-cli via a Linux instance (I'm using Amazon Linux in AWS)
• Make sure you have the key that you provided to PersistKeysToStackExchangeRedis(redis, key: [KEY]) in your Startup.cs

Migrating Keys

1. Change your Startup.cs to use MySQL for the data source and comment out/remove Redis configuration, replacing MyKeysContext with your DbContext name. In this example I'm using MyAppDataProtectionKeys.

   services.AddDataProtection()
       // .PersistKeysToStackExchangeRedis(redis, "MyAppDataProtectionKeys")
       .PersistKeysToDbContext<MyKeysContext>(); // <<-- New config
   

2. Ensure you have added the DataProtection table definitions to your DbContext and run the migrations to add the tables to MySQL, more info here: https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/implementation/key-storage-providers?view=aspnetcore-8.0&tabs=visual-studio

3. Query your Redis instance for your existing keys and run insert scripts to insert/migrate the keys from Redis to your newly created MySQL tables.

I have encapsulated the generation of the insert scripts using the following bash script. It echos the required scripts to enter data into the table, copy them from your console and execute them against your database:

#!/bin/bash

# Redis connection details
redis_host="[YOUR HOST HERE]"
redis_port=6379
redis_key="[YOUR KEY HERE]"

# MySQL details, update to your table name if required
table_name="MyAppDataProtectionKeys"

# Fetch values from Redis using LRANGE
redis_values=$(redis-cli -h $redis_host -p $redis_port LRANGE $redis_key 0 -1)

# Loop through each value and generate MySQL insert queries
IFS=$'\n' # Set Internal Field Separator to newline to handle values with spaces
for value in $redis_values; do
    unescaped_xml=$(echo -n "$value" | sed 's/\\//g')  # Unescape backslashes
    id_value=$(echo "$unescaped_xml" | awk -F'id="' '{split($2, a, "\""); print a[1]}')
    insert_query="INSERT INTO $table_name (Xml, FriendlyName) VALUES ('$unescaped_xml', 'key-$id_value');"
    echo "$insert_query"
done

After inserting you should have your data protection keys in MySQL. The default layout of the table is as shown below (Columns: Id, FriendlyName, Xml):

4. Re-deploy your app with DbContext data protection key setup in startup. Being mindful to look for ERROR messages, in particular:

Microsoft.AspNetCore.Mvc.ViewFeatures.MvcViewFeaturesLoggerExtensions.AntiforgeryTokenInvalid Microsoft.AspNetCore.Mvc.ViewFeatures.Filters.ValidateAntiforgeryTokenAuthorizationFilter Antiforgery token validation failed. The antiforgery token could not be decrypted. Microsoft.AspNetCore.Antiforgery.AntiforgeryValidationException: The antiforgery token could not be decrypted.

and the following INFO messages that may be one indication that your migration was not successful:

Microsoft.AspNetCore.Antiforgery.AntiforgeryValidationException: The antiforgery token could not be decrypted. ---> System.Security.Cryptography.CryptographicException: The key {[GUID APPEARS HERE]} was not found in the key ring. For more information go to http://aka.ms/dataprotectionwarning

The following log setup is useful for surfacing these INFO messages:

"Logging": {
    "LogLevel": {
        // ... other log level configurations ...
        "Microsoft.AspNetCore.Antiforgery": "Error",          
        "Microsoft.AspNetCore.Mvc.ViewFeatures.Filters.ValidateAntiforgeryTokenAuthorizationFilter": "Debug"
    }
},

• Once happy with the migration, you can delete your keys from Redis using the command (modifying with your data protection key):

  redis-cli -h yourhost -p 6379 DEL MyAppDataProtectionKeys
  # -- or rename --
  redis-cli -h yourhost -p 6379 RENAME MyAppDataProtectionKeys NewKey
  

Other notes:

• New data protection keys may be created in your live environment during this process, make sure to verify that you have migrated all of them across