Nested named regex groups: how to maintain the nested structure in match result?

Question

A small example:

import re

pattern = re.compile(
    r"(?P<hello>(?P<nested>hello)?(?P<other>cat)?)?(?P<world>world)?"
)
result = pattern.match("hellocat world")

print(result.groups())
print(result.groupdict() if result else "NO RESULT")

produces:

('hellocat', 'hello', 'cat', None)
{'hello': 'hellocat', 'nested': 'hello', 'other': 'cat', 'world': None}

The regex match result returns a flat dictionary, rather than a dictionary of dictionaries that would correspond with the nested structure of the regex pattern. By this I mean:

{'hello': {'nested': 'hello', 'other': 'cat'}, 'world': None}

Is there a "built-in" (i.e. something involving details of what is provided by the re module) way to access the match result that does preserve the nesting structure of the regex? By this I mean that the following are not solutions in the context of this question:

• parsing the regex pattern myself to determine nested groups
• using a data structure that represents a regex pattern as a nested structure, and then implementing logic for that data structure to match against a string as if it were a "flat" regex pattern.

Answer 1

Since you don't mind using implementation details of the re module (which are subject to undocumented future changes), what you want is then possible by overriding the hooks that are called when the parser enters and leaves a capture group.

Reading the source code of the Python implementation of re's parser we can find that it calls the opengroup method of the re._parser.State object when entering a capture group, and calls the closegroup method when leaving.

We can therefore patch State with an additional attribute of a stack of dicts representing the sub-tree of the current group, override opengroup and closegroup to build the sub-trees when entering and leaving groups, and provide a method nestedgroupdict to fill the leaves (which have empty sub-trees) with actual matching values from the output of the groupdict method of a match:

import re

class State(re._parser.State):
    def __init__(self):
        super().__init__()
        self.treestack = [{}]

    def opengroup(self, name=None):
        self.treestack[-1][name] = subtree = {}
        self.treestack.append(subtree)
        return super().opengroup(name)

    def closegroup(self, gid, p):
        self.treestack.pop()
        super().closegroup(gid, p)

    def nestedgroupdict(self, groupdict, _tree=None):
        if _tree is None:
            _tree, = self.treestack
        result = {}
        for name, subtree in _tree.items():
            if subtree:
                result[name] = self.nestedgroupdict(groupdict, subtree)
            else:
                result[name] = groupdict[name]
        return result

re._parser.State = State

so that the parser will produce a state with treestack containing a structure of the named groups:

parsed = re._parser.parse(
    r"(?P<hello>(?P<nested>hello)?(?P<other>cat)?)?(?P<world>world)?"
)
print(parsed.state.treestack)

which outputs:

[{'hello': {'nested': {}, 'other': {}}, 'world': {}}]

We can then compile the parsed pattern to match it against a string and call groupdict to get the group-value mapping to feed into our nestedgroupdict method of the state to produce the desired nested structure:

groupdict = re._compiler.compile(parsed).match("hellocat world").groupdict()
print(parsed.state.nestedgroupdict(groupdict))

which outputs:

{'hello': {'nested': 'hello', 'other': 'cat'}, 'world': None}

Demo here