Parsing Binary files that contains BCD (Binary Coded Decimal) values using Python Numpy

I have a binary file that has some fields encoded as BCD (Binary Coded Decimal). Example as below.

14 75 26 58 87 7F (Raw bytes in hex format).

I am using (np.void, 6) to read and convert from binary file and below is the output I am getting.

b'\x14\x75\x26\x58\x87\x7F'

But I would like to get the output as '14752658877', without the fill character 'F' using numpy.

Below is the code: with open (filename, "rb") as f:

    while True:

        chunk = f.read(chunksize)

        if (chunk):

            dt = np.dtype([('a','b'), ('b', '>i4'), ('c', 'S15'),('d', np.str, 7),
                                   ('e', 'S7'), ('f', np.void, 6)])

            x = np.frombuffer (chunk, dtype=dt)
            print (x)

        else:
            break

Also, the input file contains many fixed length binary records. What is the efficient way to convert and store it as ascii file using numpy.

Solution

I don't know if numpy can somehow accelerate this, but a specalized function can be quickly constructed:

fastDict = {16*(i//10)+(i%10):i for i in range(100)}

def bcdToInteger(bcd):
    result = 0
    while bcd and bcd[0] in fastDict:
        result *= 100
        result += fastDict[bcd[0]]
        bcd = bcd[1:]
    if bcd and bcd[0] & 0xf0 <= 0x90:
        result *= 10
        result += bcd[0]>>4
        if bcd[0] & 0xf <= 9:
            result *= 10
            result += bcd[0] & 0x0f
    return result

>>> print (bcdToInteger(b'\x14\x75\x26\x58\x87\x7F'))  # your sequence
14752658877
>>> print (bcdToInteger(b'\x12\x34\xA0'))   # first invalid nibble ends
1234
>>> print (bcdToInteger(b'\x00\x00\x99'))   # and so does an end of string
99
>>> print (bcdToInteger(b'\x1F'))           # a single nibble value
1

As long as you keep feeding it valid BCD bytes, it multiplies the result by 100 and adds the two new digits. Only the final byte needs some further inspection: if the highest nibble is valid, the result thus far gets multiplied by 10 and that nibble gets added. If the lowest nibble is valid as well, this is repeated.

The fastDict is to speed things up. It's a dictionary that returns the correct value for all 100 hex bytes from 00 to 99 so the number of actual calculations is as small as possible. You can do without the dictionary, but that means you have to do the comparisons and calculations in the if block for every single byte.