Search code examples
.netvb.netfilemathprogress

Report progress properly in this file operation?


This method splits a file into chunks, I would like to report the progress (the percent) of the remaining chunks and the total remaining bytes to accomplish the task.

I've messed my head with the arithmetic operations, someone could help me?

Please note in the code the instructions where I need to report the progress, I used labels just for test, I will keep the values in variables:

Label1.Text = ...

Here is the method:

    Public Sub SplitFile(ByVal InputFile As String,
                        ByVal ChunkSize As Long,
                        Optional ByVal ChunkName As String = Nothing,
                        Optional ByVal ChunkExt As String = Nothing,
                        Optional ByVal Overwrite As Boolean = False)

       ' FileInfo instance of the input file.
       Dim fInfo As New IO.FileInfo(InputFile)

       ' The buffer to read data and write the chunks.
       Dim Buffer As Byte() = New Byte() {}

       ' The buffer length.
       Dim BufferSize As Integer = 1048576I ' 1048576 = 1 mb | 33554432 = 32 mb | 67108864 = 64 mb

       ' Counts the length of the current chunk file.
       Dim BytesWritten As Long = 0L

       ' The total amount of chunks to create.
       Dim ChunkCount As Integer = CInt(Math.Floor(fInfo.Length / ChunkSize))

       ' Keeps track of the current chunk.
       Dim ChunkIndex As Integer = 0I

       ' A zero-filled string to enumerate the chunk files.
       Dim Zeros As String = String.Empty

       ' The given filename for each chunk.
       Dim ChunkFile As String = String.Empty

       ' The chunk file basename.
       ChunkName = If(String.IsNullOrEmpty(ChunkName),
                      IO.Path.Combine(fInfo.DirectoryName, IO.Path.GetFileNameWithoutExtension(fInfo.Name)),
                      IO.Path.Combine(fInfo.DirectoryName, ChunkName))

       ' The chunk file extension.
       ChunkExt = If(String.IsNullOrEmpty(ChunkExt),
                     fInfo.Extension.Substring(1I),
                     ChunkExt)

       ' If ChunkSize is bigger than filesize then...
       If ChunkSize >= fInfo.Length Then
           Throw New OverflowException("'ChunkSize' should be smaller than the Filesize.")
           Exit Sub

           ' For cases where a chunksize is smaller than the buffersize.
       ElseIf ChunkSize < BufferSize Then
           BufferSize = CInt(ChunkSize)

       End If ' ChunkSize <>...

       ' If not file-overwrite is allowed then...
       If Not Overwrite Then

           For Index As Integer = 0I To (ChunkCount)

               ' Set chunk filename.
               Zeros = New String("0", CStr(ChunkCount).Length - CStr(Index + 1I).Length)
               ChunkFile = String.Format("{0}.{1}.{2}", ChunkName, Zeros & CStr(Index + 1I), ChunkExt)

               ' If chunk file already exists then...
               If IO.File.Exists(ChunkFile) Then

                   Throw New IO.IOException(String.Format("File already exist: {0}", ChunkFile))
                   Exit Sub

               End If ' IO.File.Exists(ChunkFile)

           Next Index

           Zeros = String.Empty
           ChunkFile = String.Empty

       End If ' Overwrite

       ' Open the file to start reading bytes.
       Using InputStream As New IO.FileStream(fInfo.FullName, IO.FileMode.Open)

           Using BinaryReader As New IO.BinaryReader(InputStream)

               While (InputStream.Position < InputStream.Length)

                   ' Set chunk filename.
                   Zeros = New String("0", CStr(ChunkCount).Length - CStr(ChunkIndex + 1I).Length)
                   ChunkFile = String.Format("{0}.{1}.{2}", ChunkName, Zeros & CStr(ChunkIndex + 1I), ChunkExt)

                   ' Reset written byte-length counter.
                   BytesWritten = 0L

                   ' Create the chunk file to Write the bytes.
                   Using OutputStream As New IO.FileStream(ChunkFile, IO.FileMode.Create)

                       Using BinaryWriter As New IO.BinaryWriter(OutputStream)

                           ' Read until reached the end-bytes of the input file.
                           While (BytesWritten < ChunkSize) AndAlso (InputStream.Position < InputStream.Length)

                               ' Read bytes from the original file (BufferSize byte-length).
                               Buffer = BinaryReader.ReadBytes(BufferSize)

                               ' Write those bytes in the chunk file.
                               BinaryWriter.Write(Buffer)

                               ' Increment the size counter.
                               BytesWritten += Buffer.Count

                               ' Use the written bytes of this chunk to calculate the total remaining bytes of the operation (not only for this chunk)...
                               ' Label1.Text = CDbl((100L / ChunkSize) * BytesWritten) / (ChunkIndex / ChunkCount) '/ fInfo.Length
                               Application.DoEvents()

                           End While ' (BytesWritten < ChunkSize) AndAlso (InputStream.Position < InputStream.Length)

                           OutputStream.Flush()

                       End Using ' BinaryWriter

                   End Using ' OutputStream

                   ChunkIndex += 1I 'Increment file counter

                   ' Report progress of remaining chunks...
                   ' Label1.Text = CDbl((100I / ChunkCount) * (ChunkIndex))

               End While ' InputStream.Position < InputStream.Length

           End Using ' BinaryReader

       End Using ' InputStream

   End Sub

UPDATE:

With this I can print the progress when a chunk is created, its ok but its very unprecise progress lets say we have to split a file in chunks of 3 GB each one, then I only can report a progress-change after some time when a chunk finished to create:

Debug.WriteLine(((ChunkIndex) / ChunkCount) * 100I)

That's the reason why I'm trying to work a progress report when the bytes are written in each chunk, that way I could report a progress-change quickly every second, note this part in the code above, here is where I need help:

' Use the written bytes of this chunk to calculate the total remaining bytes of the operation (not only for this chunk)...
' Label1.Text = CDbl((100L / ChunkSize) * BytesWritten) / (ChunkIndex / ChunkCount) '/ fInfo.Length

Solution

  • A few style things:

    It's usually a good idea to name local variables in lower case. It makes it a lot easier to read and follow your code. When they begin in uppercase, they will be mistaken for method names (maybe this is just my preference)

    It's not necessary to suffix numbers with I or L. When you declare the variable, the compiler will know what to do when you explicitly specify the type.

    To answer your question:

    If I understand that you just want more granular progress data, simply declare a variable to hold the total size of the file and subtract the bytes you've written after each chunked buffer operation. Add this declaration:

        Dim bytesRemaining = fInfo.Length
    

    and then change your While loop from this:

    While (BytesWritten < ChunkSize) AndAlso (InputStream.Position < InputStream.Length)
    
        ' Read bytes from the original file (BufferSize byte-length).
        Buffer = BinaryReader.ReadBytes(BufferSize)
    
        ' Write those bytes in the chunk file.
        BinaryWriter.Write(Buffer)
    
        ' Increment the size counter.
        BytesWritten += Buffer.Count
    
        ' Use the written bytes of this chunk to calculate the total remaining bytes of the operation (not only for this chunk)...
        ' Label1.Text = CDbl((100L / ChunkSize) * BytesWritten) / (ChunkIndex / ChunkCount) '/ fInfo.Length
        Application.DoEvents()
    
    End While ' (BytesWritten < ChunkSize) AndAlso (InputStream.Position < InputStream.Length)
    

    To this:

    While (BytesWritten < ChunkSize) AndAlso (InputStream.Position < InputStream.Length)
    
        ' Read bytes from the original file (BufferSize byte-length).
        Buffer = BinaryReader.ReadBytes(BufferSize)
    
        ' Write those bytes in the chunk file.
        BinaryWriter.Write(Buffer)
    
        ' Increment the size counter.
        BytesWritten += Buffer.Count
    
        bytesRemaining -= Buffer.Count
    
        ' Use the written bytes of this chunk to calculate the total remaining bytes of the operation (not only for this chunk)...
    
        If bytesRemaining > 0 Then
            Label1.Text = (bytesRemaining / fInfo.Length) * 100
        End If
    
        Application.DoEvents()
    
    End While ' (BytesWritten < ChunkSize) AndAlso (InputStream.Position < InputStream.Length)
    

    You'll have to format the percentage how you want it.

    Also, note that I did not test this for every possible case. It's possible that on the last file something crazy will go wrong, you'll have to fix that when it happens.