Delphi: how to efficently read a big binary file, converting it to hexadecimal for passing it as a varbinary(max) parameter?

Question

I need to convert a binary file (a zip file) into hexadecimal representation, to then send it to sql-server as a varbinary(max) function parameter.

A full example (using a very small file!) is:

1) my file contains the following bits 0000111100001111

2) I need a procedure to QUICKLY convert it to 0F0F

3) I will call a sql server function passing 0x0F0F as parameter

The problem is that I have large files (up to 100MB, even if average file size is 100KB files are possible), so I need the fastest way to do this.

Otherwise stated: I need to create the string

'0x'+BinaryDataInHexadecimalRepresentation

in the most efficient way. (Note: may be there is a way to immediately open a file and obtain an hexadecimal string, so in this case all I need is to use "this way", if it is there).

Related question: passing hexadecimal data to sql server

UPDATE: after reading the comments I think it is needed to add more information here. The reason why I try to use a T-SQL text command to send the binary data to the stored procedure is that in this way I remove some overhead to the server: the stored prcoedure recieves the binary data and writes it to a file (this is my final goal). If I use a DAC component I will be able to easily send the biray data to the server, but in that case I need to use a temp table to store the data, and then sending this data to the storedprocedure that writes the file.

So the idea is:

1) using T-SQL "long" command: more overhead on client because I need to read the file and convert it to hexadecimal to preparing the long command; less server overhead since sql server just recieves the binary data and processes it in the stored function

2) using DAC: I need to pass through a temp table in sql server, therefore having more overhead on the server

Since I am using the server as web document server (it's a trick), I want to try to reduce the overhead on the server. Anyway may be I am wrong and (2) anyway is a better technique than (1)

Answer 1

Well here's a option that would do the conversion as fast as I can think of.

Features of the code:

• Only one allocation for the string (so no realloc and no move or copy)
• Fast read from the file.

Since we know that one byte translates to exactly two hexadecimal chars, we know our result string needs to be exactly twice the size of the file. We allocate an string of the required size and then we read from the file in large-enough blocks so the OS can optimize it for us (reading byte-by-byte is evil). We use an actual string but we write into the string using an pointer:

function TEditorDeschidereDeCredite.FileToHex(FileName: string): AnsiString;
var FS:TFileStream;
    PS:PAnsiChar;
    Block:array[0..1023] of byte; // 1Kb
    len,i,pos:Integer;
    B:Byte;
const Nibbs: array[0..15] of AnsiChar = ('0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F');
begin
  FS := TFileStream.Create(FileName, fmOpenRead);
  try
    Result := '';
    SetLength(Result, FS.Size * 2);
    PS := PAnsiChar(Result);
    pos := 0; // position into the result string
    len := FS.Read(Block, SizeOf(Block));
    while len <> 0 do
    begin
      for i:=0 to len-1 do
      begin
        B := Block[i];
        PS[pos] := Nibbs[B div $F];
        Inc(pos);
        PS[pos] := Nibbs[B mod $F];
        Inc(pos);
      end;
      len := FS.Read(Block, SizeOf(Block));
    end;
  finally FS.Free;
  end;
end;

P.S: I'm using AnsiString, and PAnsiChar so the code works also works with Unicode Delphi. If you happen to be on Delphi 2010 find a way to use this in it's current form (AnsiString) so you can skip the conversions.