Storing Historical Data using History table in ASP.Net Core

Question

I have an entity in my database linked up with multiple entities and I need to store historical data for the changes (insert ,updates and delete) in the base entity as well as children entities.

Now the approach we are thinking of adopting is to keep every data in the corresponding history entity. The base table looks like this :

public partial class Con 
{
    public Guid Id { get; set; }
    public string Note { get; set; }
    ...
    public virtual ICollection<Document> Document { get; set; }
    public virtual ICollection<ConLine> ConLine { get; set; }
    public virtual ICollection<Leg> Leg { get; set; }
}

And the historical table looks like this, I am not sure about how to design the history table to hold linked tables data:

public partial class ConHistory 
{
    public Guid Id { get; set; }
    public Guid ConId { get; set; }
    public int TransactionId { get; set; }
    ...
    public virtual ICollection<Document> Document { get; set; }
    public virtual ICollection<ConLine> ConLine { get; set; }
    public virtual ICollection<Leg> Leg { get; set; }
}

How do I approach this problem and What are the best industry practices? My main concern is when a child's data is being changed how do I log in to parent history table and the corresponding child history table.

Answer 1

For simple time-series data, keeping a separate table with a copy of the modifiable fields is a perfectly valid approach. Unfortunately, in your case you would also need to make a copy of each of the linked tables as well, so that you can maintain consistent foreign keys - e.g. DocumentHistory, ConLineHistory, LegHistory. It's a lot of repetitive code. And then you have to consider, what happens to all the historical records when the schema changes?

Personally, I would store this information as json, in a text column. All the fields that you search on should be in sql, so that you can index it, but the rest can be serialized to a json string:

public partial class ConHistory 
{
    public Guid Id { get; set; }
    public Guid ConId { get; set; }
    public int TransactionId { get; set; }
    public Guid ModifiedByUser { get; set; }
    public DateTime Date { get; set; }

    // Serialize the rest of the `ConHistory` fields to a json object, and store them here
    public string Json { get; set; }
}

Sql also has the JSON_VALUE function, and there are some examples of using this in Entity Framework, if you do need to get values out of the json string for querying.