Parsing Yann LeCun's MNIST IDX file format

Question

I would like to understand how to open this version of the MNIST data set. For example, the training set label file train-labels-idx1-ubyte is defined as:

TRAINING SET LABEL FILE (train-labels-idx1-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  60000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
........
xxxx     unsigned byte   ??               label

And I found some code online that seems to work, but do not understand how it works:

with open('train-labels-idx1-ubyte', 'rb') as f:
    bytes = f.read(8)
    magic, size = struct.unpack(">II", bytes)

print(magic) # 2049
print(size)  # 60000

My understanding is that struct.unpack interprets the second argument as a big-endian byte string of two 4-byte integers (See here). When I actually print the value of bytes, though, I get:

b'\x00\x00\x08\x01\x00\x00\xea`'

The first four-byte integer makes sense:

b'\x00\x00\x08\x01'

The first two bytes are 0. The next indicates the data are unsigned bytes. And 0x01 indicates a 1-dimensional vector of labels. Assuming my understanding is correct so far, what is happening with the next three (four?) bytes:

...\x00\x00\xea`

How does this translate to 60,000?

Answer 1

I wrote the following code in case anyone needs to parse the whole dataset of images (as it appears in the question's title), and not just the first two bytes.

import numpy as np
import struct

with open('samples/t10k-images-idx3-ubyte','rb') as f:
    magic, size = struct.unpack(">II", f.read(8))
    nrows, ncols = struct.unpack(">II", f.read(8))
    data = np.fromfile(f, dtype=np.dtype(np.uint8).newbyteorder('>'))
    data = data.reshape((size, nrows, ncols))

This assumes you uncompressed the .gz file. You can also work with the compressed file, as indicated by Marktodisco's answer, by adding import gzip, using gzip.open(...) instead of open(...), and using np.frombuffer(f.read(), ...) instead of np.fromfile(f, ...).

And just to check, show the first digit. In my case it's a 7.

import matplotlib.pyplot as plt
plt.imshow(data[0,:,:], cmap='gray')
plt.show()

In addition, the following code reads the file with labels

with open('samples/t10k-labels-idx1-ubyte','rb') as f:
    magic, size = struct.unpack(">II", f.read(8))
    data = np.fromfile(f, dtype=np.dtype(np.uint8).newbyteorder('>'))
    data = data.reshape((size,)) # (Optional)
print(data)
# Prints: [7 2 1 ... 4 5 6]

The last reshape can be (size,) or (1, size) depending on your standards.