Splitting huge sql dump file in smallest chunks possible

Question

I'm trying to upload huge SQL files (over a couple of GB) to a remote mysql server using C#.

Is there any standard way to;

1. Start reading an SQL file to memory
2. Stop once we have something usable
3. Execute that bit
4. Remove that bit from memory
5. Read to the next usable part
6. Etc.

Or do I have to write a regex or something myself? The files are too big to read to memory in one go.

Start of one of the sql files (if it helps at all)

-- MySQL dump 10.13  Distrib 5.1.49, for debian-linux-gnu (x86_64)
--
-- Host: localhost    Database: qqqq
-- ------------------------------------------------------
-- Server version   5.1.49-3

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;

--
-- Current Database: `qqqq`
--

CREATE DATABASE /*!32312 IF NOT EXISTS*/ `qqqq` /*!40100 DEFAULT CHARACTER SET latin1 */;

USE `qqqq`;

--
-- Table structure for table `config`
--

DROP TABLE IF EXISTS `config`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `config` (
  `name` varchar(200) NOT NULL,
  `value` varchar(200) NOT NULL,
  `timestamp` int(11) NOT NULL,
  UNIQUE KEY `name` (`name`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `config`
--

LOCK TABLES `config` WRITE;
/*!40000 ALTER TABLE `config` DISABLE KEYS */;
INSERT INTO `config` VALUES ('someConfigValue','324',0),('someConfigValue2','27',0),('someConfigValue3','0',0);
/*!40000 ALTER TABLE `config` ENABLE KEYS */;
UNLOCK TABLES;

I think I can't just split it each time a ; occurs, because one of those may be inside a text string as well.

Answer 1

Try looking for common break points, or similar places where it would be safe to split the SQL file. For example:

--
-- (command here)
--

Seems to be used to denote an 'important' command, which comes before a block of operations. You could try splitting it for every --\n-- as a common 'safespot', as line breaks should be escaped inside queries.