PyYAML alternative constructors for standard tags

In the attempt of creating a loader that is capable of expanding variables, I noticed that PyYAML was not expanding the variables when I used block scalars (e.g. >-). So, take this example:

# ademo.yml
key1: foo
key2: bar $LOL bar
key3: >-
    baz $ASD baz

To expand the variables I did the following:

# yaamer.py
import re
import sys
import yaml

# I create a loder
class MyLoader(yaml.SafeLoader): pass

# Add to it an implicit resolver that matches variables
matcher = re.compile(r".*\$[A-Z].*")
MyLoader.add_implicit_resolver("!path", matcher, None)

# Add a constructor that would replaces the variable
def cons(loader, node):
    print("Constructing for", node)
    return "FOO"
MyLoader.add_constructor("!path", cons)

with open(sys.argv[1]) as inf:
    print(yaml.load(inf, MyLoader))

When I call this, python yaamer.py ademo.yml, I get this output:

Constructing for ScalarNode(tag='!path', value='bar $LOL bar')
{'key1': 'foo', 'key2': 'FOO', 'key3': 'baz $ASD baz'}

So, the replacement is just in key2, but not key3.

To understand what was happening I implemented a wrapper for the regex that lets me understand what's going on:

class Matcher:
    def __init__(self):
        self.matcher = re.compile(r".*\$[A-Z].*")                                                                 
    def match(self, *args, **kwargs):
        print("Matching", args, kwargs)
        return self.matcher.match(*args, **kwargs)

and using it leads to this output:

Matching ('key1',) {}
Matching ('foo',) {}
Matching ('key2',) {}
Matching ('bar $LOL bar',) {}
Matching ('key3',) {}
Constructing for ScalarNode(tag='!path', value='bar $LOL bar')
{'key1': 'foo', 'key2': 'FOO', 'key3': 'baz $ASD baz'}

There is no trace of the block scalar in the match! So, apparently, this matcher is never used for parsing >- blocks.

So I investigated and found out that there are some pre-defined tags, such as !!str or !!map. I though that, since >- is a special syntax, it would be automatically matched to one of those tags.

So I proceeded installing custom constructors for all the tags, to see if any of those was called:

def make_cons(name):
    def _cons(loader, node):
        print("Constructor for", name)
        return node.value
    return _cons

MyLoader.add_constructor("!!str", make_cons('!!str'))
MyLoader.add_constructor("!!int", make_cons('!!int'))

But they are never called! What is going on?? If I had to install custom constructors for pre-existing YAML tags, how would I do that??

Solution

An implicit resolver can only match plain scalars, not quoted. Quoted means, single or double quoted, or a literal or folded block scalar. So your key3 is a folded block scalar and therefor not considered for matching.

You can explicitly add the !path tag to it in the YAML source, then your custom constructor will be called.

See the PyYAML Documentation and look for the words 'implicit' and 'plain'.

To add constructors for the standard tags, you need to use the full name. !!str is just a shorthand for tag:yaml.org,2002:str:

yaml.add_constructor('tag:yaml.org,2002:str', my_constructor)