Search code examples
vb.netencryptionvb6visual-studio-2017aes

copymemory VB6/VBA to VB.net Arithmetic operation resulted in an overflow


Hey all I am trying to figure out how to get the following to work or replace in order to make work:

Module1:

Private oTest                   As Class1
Private InitDone                As Boolean
Private Map1(0 To 63)           As Byte
Private Map2(0 To 127)          As Byte

#If VBA7 Then
  Public Declare PtrSafe Function GetSystemMetrics Lib "user32" (ByVal nIndex As Long) As Long
#Else
  Public Declare Function GetSystemMetrics Lib "user32" (ByVal nIndex As Long) As Long
#End If

Private Declare Sub CopyMemoryByref Lib "Kernel32.dll" & _
                    Alias "RtlMoveMemory" (ByRef dest As Integer, ByRef & _
                    source As Integer, ByVal numBytes As Integer)

Private Declare Function VarPtr Lib "vb40032.dll" & _
                         Alias "VarPtr" (lpObject As Integer) As Long

Public Function EncryptData(ByRef bytMessage() As Byte, ByRef bytPassword() As Byte) As Byte()
        Dim bytKey(31) As Byte
        Dim bytIn() As Byte
        Dim bytOut() As Byte
        Dim bytTemp(31) As Byte
        Dim lCount, lLength As Integer
        Dim lEncodedLength, lPosition As Integer
        Dim bytLen(3) As Byte

        If Not IsInitialized(bytMessage) Then Exit Function
        If Not IsInitialized(bytPassword) Then Exit Function

        For lCount = 0 To UBound(bytPassword)
            bytKey(lCount) = bytPassword(lCount) : If lCount = 31 Then Exit For
        Next lCount

        gentables()
        gkey(8, 8, bytKey)
        lLength = UBound(bytMessage) + 1 : lEncodedLength = lLength + 4

        If lEncodedLength Mod 32 <> 0 Then lEncodedLength = lEncodedLength + 32 - (lEncodedLength Mod 32)

        ReDim bytIn(lEncodedLength - 1) : ReDim bytOut(lEncodedLength - 1)

        Try
            CopyMemory(VarPtr(bytIn(0)), VarPtr(lLength), 4)
        Catch ex As Exception
            MessageBox.Show(ex.Message)
        End Try

        For lCount = 0 To lEncodedLength - 1 Step 32
           CopyMemory(VarPtr(bytTemp(0)), VarPtr(bytIn(lCount)), 32)
           Encrypt(bytTemp)
           CopyMemory(VarPtr(bytOut(lCount)), VarPtr(bytTemp(0)), 32)
        Next lCount
End Function

UserForm:

Private Sub Command1_Click(ByVal eventSender As System.Object, ByVal eventArgs As System.EventArgs) Handles Command1.Click
    Dim sTemp, sPassword As String

    sTemp = "this is a test"
    Debug.Print("       To encrypt: " & sTemp)

    sPassword = "blk@vjdii:5@fAB5"
    Debug.Print(StrReverse(sPassword))

    sPassword = Str2Hex(strEncrypt(sPassword, StrReverse(sPassword)))
    Debug.Print("       Secret key: " & sPassword)

    sTemp = Str2Hex(strEncrypt(sTemp, sPassword))
    Debug.Print("          Encrypt: " & sTemp)

    sTemp = Base64EncodeString(sTemp)
    Debug.Print("Encrypt w/ Base64: " & sTemp)

    sTemp = Base64DecodeString(sTemp)
    Debug.Print("Decrypt w/ Base64: " & sTemp)

    sTemp = Hex2Str(sTemp)
    Debug.Print("          Decrypt: " & strDecrypt(sTemp, sPassword))
End Sub

 This code works just fine when using it within the VBA excel code. This is the VB6 output:

To encrypt: this is a test

Secret key: F050C1C2B61E8DCC349DC498D9993F8D11330F12D9E0071B4B83D172FEBE5AED

Encrypt: F899ABA853D21B20F889CFD18BB42C472187B4E1CF613139370313DFD8A492DE

Encrypt w/ Base64:

Rjg5OUFCQTg1M0QyMUIyMEY4ODlDRkQxOEJCNDJDNDcyMTg3QjRFMUNGNjEzMTM5MzcwMzEzREZEOEE 0OTJERQ==

Decrypt w/ Base64: F899ABA853D21B20F889CFD18BB42C472187B4E1CF613139370313DFD8A492DE

Decrypt: this is a test

However, when converting it over to .net I get the error of:

Arithmetic operation resulted in an overflow.

on the line CopyMemory(VarPtr(bytIn(0)), VarPtr(lLength), 4).

How can I re-write this in order for it to work so both VB.net and VB6 can share the code base in order to encrypt/decrypt string messages back and forth?


Solution

  • Use the Array.Copy Method in VB.NET. VB.NET has a completely different memory model than VB6. You cannot apply VB6 functions from VB6 DLLs in VB.NET! .NET has its own encryption system. See: System.Security.Cryptography Namespace

    Array.Copy(bytTemp, 0, bytIn, lCount, 32)
    

    VB.NET is not just a VB7. VB.NET is a completely new language with a completely new type system, new libraries, new DLL and EXE structures, new runtime infrastructure, new ... (well almost everything is new)

    As Hans Passant already hinted at: it is almost impossble to have a common code base for VB6 and VB.NET