Reading changes in a file in real-time using .NET

Question

I have a .csv file that is frequently updated (about 20 to 30 times per minute). I want to insert the newly added lines to a database as soon as they are written to the file.

The FileSystemWatcher class listens to the file system change notifications and can raise an event whenever there is a change in a specified file. The problem is that the FileSystemWatcher cannot determine exactly which lines were added or removed (as far as I know).

One way to read those lines is to save and compare the line count between changes and read the difference between the last and second last change. However, I am looking for a cleaner (perhaps more elegant) solution.

Answer 1

I've written something very similar. I used the FileSystemWatcher to get notifications about changes. I then used a FileStream to read the data (keeping track of my last position within the file and seeking to that before reading the new data). Then I add the read data to a buffer which automatically extracts complete lines and then outputs then to the UI.

Note: "this.MoreData(..) is an event, the listener of which adds to the aforementioned buffer, and handles the complete line extraction.

Note: As has already been mentioned, this will only work if the modifications are always additions to the file. Any deletions will cause problems.

Hope this helps.

   public void File_Changed( object source, FileSystemEventArgs e )
    {
        lock ( this )
        {
            if ( !this.bPaused )
            {
                bool bMoreData = false;

                // Read from current seek position to end of file
                byte[] bytesRead = new byte[this.iMaxBytes];
                FileStream fs = new FileStream( this.strFilename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite );

                if ( 0 == this.iPreviousSeekPos )
                {
                    if ( this.bReadFromStart )
                    {
                        if ( null != this.BeginReadStart )
                        {
                            this.BeginReadStart( null, null );
                        }
                        this.bReadingFromStart = true;
                    }
                    else
                    {
                        if ( fs.Length > this.iMaxBytes )
                        {
                            this.iPreviousSeekPos = fs.Length - this.iMaxBytes;
                        }
                    }
                }

                this.iPreviousSeekPos = (int)fs.Seek( this.iPreviousSeekPos, SeekOrigin.Begin );
                int iNumBytes = fs.Read( bytesRead, 0, this.iMaxBytes );
                this.iPreviousSeekPos += iNumBytes;

                // If we haven't read all the data, then raise another event
                if ( this.iPreviousSeekPos < fs.Length )
                {
                    bMoreData = true;
                }

                fs.Close();

                string strData = this.encoding.GetString( bytesRead );
                this.MoreData( this, strData );

                if ( bMoreData )
                {
                    File_Changed( null, null );
                }
                else
                {
                    if ( this.bReadingFromStart )
                    {
                        this.bReadingFromStart = false;
                        if ( null != this.EndReadStart )
                        {
                            this.EndReadStart( null, null );
                        }
                    }
                }
            }
        }