Dictionary of (named) tuples in Python and speed/RAM performance

I'm creating a dictionary d of one million of items which are tuples, and ideally I'd like to access them with:

d[1634].id       # or  d[1634]['id']
d[1634].name     # or  d[1634]['name']
d[1634].isvalid  # or  d[1634]['isvalid']

rather than d[1634][0], d[1634][1], d[1634][2] which is less explicit.

According to my test:

import os, psutil, time, collections, typing
Tri = collections.namedtuple('Tri', 'id,name,isvalid')
Tri2 = typing.NamedTuple("Tri2", [('id', int), ('name', str), ('isvalid', bool)])
t0 = time.time()
# uncomment only one of these 4 next lines:
d = {i: (i+1, 'hello', True) for i in range(1000000)}                                 # tuple
# d = {i: {'id': i+1, 'name': 'hello', 'isvalid': True} for i in range(1000000)}      # dict
# d = {i: Tri(id=i+1, name='hello', isvalid=True) for i in range(1000000)}            # namedtuple
# d = {i: Tri2(id=i+1, name='hello', isvalid=True) for i in range(1000000)}            # NamedTuple
print('%.3f s  %.1f MB' % (time.time()-t0, psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2))

"""
tuple:       0.257 s  193.3 MB
dict:        0.329 s  363.6 MB
namedtuple:  1.253 s  193.3 MB  (collections)
NamedTuple:  1.250 s  193.5 MB  (typing)
"""

using a dict doubles the RAM usage, compared to a tuple
using a namedtuple or NamedTuple multiplies by 5 the time spent, compared to a tuple!

Question: is there a tuple-like data structure in Python 3 which allows to access the data with x.id, x.name, etc. and also is RAM and CPU efficient?

Notes:

in my real use case, the tuple is something like a C-struct of type (uint64, uint64, bool).
I've also tried with:
- slots (to avoid the interal object's __dict__, see Usage of __slots__?)
- dataclass:
```
@dataclasses.dataclass
class Tri3:
    id: int
    ...
```
- ctypes.Structure:
```
class Tri7(ctypes.Structure):
    _fields_ = [("id", ctypes.c_int), ...]
```
but it was not better (all of them ~ 1.2 sec.), nothing close to a genuine tuple in terms of performance
Here are other options: C-like structures in Python

Solution

Cython's cdef-classes might be what you want: They use less memory than the pure Python classes, even if it comes at costs of more overhead when accessing members (because fields are stored as C-values and not Python-objects).

For example:

%%cython
cdef class CTuple:
    cdef public unsigned long long int id
    cdef public str name
    cdef public bint isvalid
    
    def __init__(self, id, name, isvalid):
        self.id = id
        self.name = name
        self.isvalid = isvalid

which can be used as wished:

ob=CTuple(1,"mmm",3)
ob.id, ob.name, ob.isvalid # prints (2, "mmm", 3)

Timings/memory consumption:

First, the baseline on my machine:

0.258 s  252.4 MB  # tuples
0.343 s  417.5 MB  # dict
1.181 s  264.0 MB  # namedtuple collections

with CTuple we get:

0.306 s  191.0 MB

which is almost as fast and needs considerable less memory.

If the C-type of members isn't clear at compile time, one could use simple python-objects:

%%cython
cdef class PTuple:
    cdef public object id
    cdef public object name
    cdef public object isvalid
    
    def __init__(self, id, name, isvalid):
        self.id = id
        self.name = name
        self.isvalid = isvalid

The timings are a little bit surprising:

0.648 s  249.8 MB

I didn't expect it to be so much slower than the CTuple-version, but at least it is twice as fast as named tuples.

One disadvantage of this approach is that it needs compilation. Cython however offers cython.inline which can be used to compile Cython-code created on-the-fly.

I've released cynamedtuple which can be installed via pip install cynamedtuple, and is based on the prototype bellow:

import cython

# for generation of cython code:
tab = "    "
def create_members_definition(name_to_ctype):
    members = []
    for my_name, my_ctype in name_to_ctype.items():
        members.append(tab+"cdef public "+my_ctype+" "+my_name)
    return members

def create_signature(names):
    return tab + "def __init__(self,"+", ".join(names)+"):"

def create_initialization(names):
    inits = [tab+tab+"self."+x+" = "+x for x in names]
    return inits

def create_cdef_class_code(classname, names):
    code_lines = ["cdef class " + classname + ":"]
    code_lines.extend(create_members_definition(names))
    code_lines.append(create_signature(names.keys()))
    code_lines.extend(create_initialization(names.keys()))
    return "\n".join(code_lines)+"\n"

# utilize cython.inline to generate and load pyx-module:
def create_cnamedtuple_class(classname, names):
    code = create_cdef_class_code(classname, names)
    code = code + "GenericClass = " + classname +"\n"
    ret = cython.inline(code)
    return ret["GenericClass"]

Which can be used as follows, to dynamically define CTuple from above:

CTuple = create_cnamedtuple_class("CTuple", 
                                 {"id":"unsigned long long int", 
                                  "name":"str",
                                  "isvalid":"bint"})

ob = CTuple(1,"mmm",3)
...

Another alternative could be to use jit-compilation and Numba's jitted-classes which offer this possibility. They however seem to be much slower:

from numba import jitclass, types

spec = [
    ('id', types.uint64), 
    ('name', types.string),
    ('isvalid',  types.uint8),
]

@jitclass(spec)
class NBTuple(object):
    def __init__(self, id, name, isvalid):
        self.id = id
        self.name = name
        self.isvalid = isvalid

and the results are:

20.622 s  394.0 MB

so numba jitted classes are not (yet?) a good choice.