Search code examples
mysqldatabase-performancebulk-load

How can I write to MYSQL DB as if i were writing to a regular file


I want to write to MYSQL DB table with the same speed as if i were writing to a regular binary file with ofstream for example.

Normally, Writing 100M bytes to a file using ofstream takes less than 1 second.

While in MYSQL, I inserted these 100M bytes in more than 9 seconds which is very long time ( I used MYISAM table and multiple-row insert statements).

In other words, I want to configure mysql such that it makes no (checks,parsing,....etc), just writing to the table as if it was a binary file

So, Is there any method to achieve that?

Edit 1:

Here is the table I want to populate,

create table Frames
(
f1 varchar(8) not null , 
f2 varchar(6) not null , 
f3 varchar(6) not null , 
f4 varchar(2) not null ,
f5 varchar(4),
f6 varchar(16000),
f7 varchar(4) not null
);

Edit 2:

Here is a snippet for the code used for insertion where MAX is defined to 10000

for (int c = 0; c < 100; ++c)
{
  string query("insert into Frames (f1,f2,f3,f4,f5,f6,f7) values");
  for (int i = 1; i <= MAX; ++i)
  {
     query += "('AAAAAAAA','000001','000002','05','5005','000000000100000000010000000001000000000100','MKJE')";
     query += (i==MAX) ? ";" : "," ;
  }
  if (mysql_query(conn, query.c_str() )) {
     fprintf(stderr, "%s\n", mysql_error(conn));
     return(1);
  }
} 

Thanks


Solution

  • You can use various tricks for improving the speed of INSERT. That link is to the MySQL manual, which covers a bunch of tips.

    But you'll never get close to the speed of writing directly to a file. MySQL is doing a lot of extra work behind the scenes, to parse SQL, enforce access privileges, enforce data constraints, and convert the data into compact representations of data types, and then organize the data into the file efficiently, and update indexes.

    If you use InnoDB (the default) as you storage engine, you always have at least the primary key index to update. InnoDB stores table data as a clustered index.

    The only way you can truly get the same speed as if you were appending to a file is by using the CSV storage engine, and append directly to that file. That is, don't use SQL at all, just write to the data file. If the file is in the right location and has the right name, it will be the data for your MySQL table.