I am parsing very large files (Unicode - Delphi 2009), and I have a very efficient routine for doing so using PChar variables as outlined in the Stackoverflow question: What is the fastest way to Parse a line in Delphi?
Everything was working great until I ran into a file that had some embedded hex:00 characters in it. This character signals the end of a PChar string and my parsing stops at that point.
However, when you load the file, as in:
FileStream := TFileStream.Create(Filename, fmOpenRead or fmShareDenyWrite);
Size := FileStream.Size;
then you find that the size of the file is much larger. If you open the file with Notepad, it loads to the end of the file, not stopping at the first hex:00 as the PChar does.
How can I read to the end of the file while still using PChar parsing without slowing down my reading/parsing too much?
The accepted code in your other question is breaking out when it reaches a #0 character. To fix that you just need to save the length of the input and check that instead. The updated code would look something like this:
type
TLexer = class
private
FData: string;
FTokenStart: PChar;
FCurrPos: PChar;
FEndPos: PChar; // << New
function GetCurrentToken: string;
public
constructor Create(const AData: string);
function GetNextToken: Boolean;
property CurrentToken: string read GetCurrentToken;
end;
{ TLexer }
constructor TLexer.Create(const AData: string);
begin
FData := AData;
FCurrPos := PChar(FData);
FEndPos := FCurrPos + Length(AData); // << New
end;
function TLexer.GetCurrentToken: string;
begin
SetString(Result, FTokenStart, FCurrPos - FTokenStart);
end;
function TLexer.GetNextToken: Boolean;
var
cp: PChar;
begin
cp := FCurrPos; // copy to local to permit register allocation
// skip whitespace
while (cp <> FEndPos) and (cp^ <= #32) do // << Changed
Inc(cp);
// terminate at end of input
Result := cp <> FEndPos; // << Changed
if Result then
begin
FTokenStart := cp;
Inc(cp);
while (cp <> FEndPos) and (cp^ > #32) do // << Changed
Inc(cp);
end;
FCurrPos := cp;
end;