I am trying to parse and analyse log files by importing some large CSV files with millions of lines in to SQLite.
I was using my .NET program to achieve the same thing on smaller log files through LINQ with out any database.
But I was facing performance and scalability issues.
Then I found SQLite might be the best option to offload performance issues from my .NET program.
I imported the log files using the following commands from SQLite command line.
(Names are replaced to just avoid providing unnecessary information):
sqlite> Create Table MLogTable(DateTimeC text, Field1 text, Field2 text, Field3 text, Field4 text, Field5 text, Field6 text, Field7 text, Field8 text, Field9 text, Field10 text);
sqlite> .separator ","
sqlite> .import "D:\\logs\\xyz\\abc.log" MLogTable
The logging mechanism seems to have a problem and i get some incomplete lines in the Log file. For example:
expected format is (f1, f2, f3 f4, f5, f6, f7, f8, f9 ,f10, f11)
but actual format of few lines is (f1, f2)
So the log file has few lines(out of millions of lines) not as expected and on import it results the following kind of error:
Error: D:\logs\xyz\abc.log line 4499214: expected 11 columns of data but found 2
The number of erroneous lines is very less and i have no control over how it can be corrected while logging.
So those lines need to be ignored while SQLite import is done.
Is it possible with SQLite?
The CSV import function of the sqlite3 command-line tool works only with well-formed CSV files.
You have to convert the file manually, or write your own importer.