Designing a logging system

I have a program that needs to process files at batch. Instead of showing errors in message boxes on screen (which will pause the execution of the program) I need to show those error messages in a Log that the user can see as the program executes.

So I DON'T need a program-execution log like this Which logging library is better?

I am using now something derived from TRichEdit. Basically, a rich edit with few extra methods like AddError(s), AddWarn(s), AddVerbose(s), etc.

  TRichLog = class(TMyRichEdit)
   private
   protected
     Indent: Integer;   { Indent new added lines x spaces }
   public
     constructor Create(AOwner: TComponent);   override;
     procedure AddBold     (CONST Mesaj: string);
     procedure AddMsg      (CONST Mesaj: string);
     procedure AddMsgLvl   (CONST Mesaj: string; MsgType: Integer);
     procedure AddColorMsg (CONST Mesaj: string; Culoare: TColor);
     procedure AddVerb     (CONST Mesaj: string);
     procedure AddHint     (CONST Mesaj: string);
     procedure AddInfo     (CONST Mesaj: string);
     procedure AddPath     (CONST Mesaj: string);
     procedure AddWarn     (CONST Mesaj: string);
     procedure AddError    (CONST Mesaj: string);
     procedure AddMsgInt   (CONST Mesaj: string; i: Integer);          { Adds a message text followed by an integer }
     procedure AddInteger  (CONST i: Integer);
     procedure AddFromFile (CONST FileName: string; MsgType: Integer);                             { Reads lines from the specified file and adds them to the log using the specified color. }
     procedure AddEmptyRow;
     procedure AddDateStamp;

     procedure Append (RamLog: TObject);       { RamLog will be typecased to TRamLog }
     procedure SaveAsRtf   (CONST FullPath: string);
     procedure AppendToFile(CONST FullPath: string);
     function  VerbosityAsString: string;
   published
     property InsertTime: Boolean      read FInsertTime write FInsertTime default FALSE;
     property InsertDate: Boolean      read FInsertDate write FInsertDate default FALSE;
     property AutoScroll: Boolean      read FAutoScroll write FAutoScroll default TRUE;            { Automatically scroll to show the last line }
     property Verbosity : Integer      read FVerbosity  write FVerbosity  default vHints;

     property OnLogError: TNotifyEvent read FLogError   write FLogError;                           { Can be used to inform the application to automatically switch to log when an error is listed }
     property OnLogWarn : TNotifyEvent read FLogWarn    write FLogWarn;

But I would like to let user dynamically filter the context. For example the user should be able to hide all Verbose messages and keep only the warnings and errors. And if the user changes his mind, to put back the verbose messages.

Can the (existing) text in RichEdit be filtered this way? If not, I would like to get some pointers about how to reimplement it. I am thinking about writing my own format to keep the lines/messages. For example:

Cannot open file,#msgErr,#Bold

The I would have a TStringGrid to display only a limited number of lines (the ones that are visible on screen). This way I could have millions of lines without actually rendering all of them on screen. Time wasted parsing won't matter since I only have to parse the visible lines.

Requirements:

Support for colors as in RichEdit (red for errors, etc)
Lightweight
Should have two classes: a visual one (based on TStringGrid) and one non-visual for console programs (log to RAM and save the log later or display the messages as simple text in the console).
No hard dependencies (Delphi edition, Indy, DB engines, 3rd party controls etc)
Small (TRichEdit increases the size of the EXE file quite a lot)
One PAS file

An alternative would be not to use the Grid and to draw the text myself (for example in a TPanel-derived component). Or maybe such control already exists.

Any constructive critics of my ideas would be welcome. Do you have a better idea than using a Grid?

Solution

IMHO we may make a difference between:

Low-level logging, which targets developers and support;
High-level logging, which targets end-users.

From your comments, sounds like if you need the 2nd kind, which is usually also called "Audit Trail", especially in terms or regulation.

We usually implement high-level "Audit Trail" by storing the events in a database. For instance, a local high-performance SQLite3 database, or a MongoDB centralized instance.

Using a RDBMS (or NoSQL DB) has several advantages:

Its structure could be at the same time fixed (e.g. by defining categories), and evolutive (via some text fields, or even some foreign keys with other tables);
It is easy to interface with your existing UI, e.g. using powerful third-party grids, with sorting and filtering abilities;
You could use SQL to search within the log content, then even define a dedicated UI for most useful cases;
It is easy to maintain (a DELETE FROM ... would get rid of older undeded entries, and potentially keeping errors in the database).

We usually do this on production, using our Open Source SOA framework: all service calls can be directly written in a SQlite3 or MongoDB instance, without any code to write! Then you could even search within the parameters using JSON queries. And you still have integrated low-level logging available.