Search code examples
c#winapiwmievent-log

High Performance Event Log

So I've been trying various ways to get Event Log data in bulk (1000+ records/second).

I need something that can filter out old logs, right now I store the last recorded event record ID and retrieve all records where the event ID is greater than that....

I've tried EventLogQuery/EventLogReader, this works fast except when I want to pull message data, in order to get a formatted message for security logs I need to call EventLogRecord.FormattedMessage(), this brings my log speed to about 150/second with easy to format logs, even worse with complicated ones.

I've tried System.Diagnoistics.EventLog, this doesn't allow me to build filters, so every time I run this it must load ALL event logs, then I can parse off any duplicates (from the last scan). I have a sever that has 200k event logs over the past two days, memory usage gets terrible due to this, so that is a no-go.

I've tried WMI using System.Management.ManagementObjectCollection, this has filtering and can pull message data from the security event log FAST (approaching ~1000/second), however it will go to about 50/60k and start to drag it's feet, down to doing about 1-2/second, eventually I'll get a Quota Violation error. :(

So either:

Is there a way to avoid the quota violation error, or do I want to use some other method for pulling event logs at this speed?

Edit:

I wrote a blog post detailing what I've learned about this:

http://www.roushtech.net/2013/10/30/high-performance-event-log-reading/

Mostly: WINAPI is your best bet, either write C++/CLR or use PInvoke.

Solution

  • I found that managed code was too slow and I ended up using the win32 APIs to retrieve the event logs (local and/or remote). They are easy enough to implement and there are a lot of examples out there of how to do it. It will also be much faster than anything you try and do with xml.

    New API: http://msdn.microsoft.com/en-us/library/aa385780(v=vs.85).aspx

    Old API: http://msdn.microsoft.com/en-us/library/aa363652(v=vs.85).aspx

    I also found that the pre-vista API worked fine for what I needed and actually performed better than the new API on post-vista machines.