Preprocess (populate, generate) files with the content of other files

Question

I'm going to explain what my problem is using a representative example.

Let's say I have these 2 configuration files:

# product-conf.file
seo_title: general_title
seo_description: seo_description
seo_canonical: seo_canonical

product_id: general_id
title: general_title
intro: general_intro

.

# service-conf.file
seo_title: general_title
seo_description: seo_description
seo_canonical: seo_canonical

service_id: general_id
title: general_title
products: list_products

As you can see, the 3 first lines (configuration fields) are exactly the same. I'm actually using YAML for these files. I would like to have pieces of code in maintainable files and include them with calls. I need a preprocessor for these. Something like:

# snippet-seo.file
seo_title: general_title
seo_description: seo_description
seo_canonical: seo_canonical

.

# product-conf-master.file

@include snippet-seo

product_id: general_id
title: general_title
intro: general_intro

.

# service-conf-master.file

@include snippet-seo

service_id: general_id
title: general_title
products: list_products

The preprocessor will read all the master files in /masters/*, attend all the calls and substituting them for the appropriate snippet from /snippets/ and saving the result in /

I'm doing the call with @ but I can choose whatever other format is suitable for the chosen preprocessor. I used this way because it strongly resembles to the SASS directive @extend or @include.

What is the best and easiest way I can achieve this? A package for node would be my first choice.

Answer 1

If you don't necessarily need to pre-process the files you can solve this with a small YAML processing program (you might be able to do this using some node based programming).

import pathlib
import ruamel.yaml

yaml = ruamel.yaml.YAML()
master = pathlib.Path('master.file')
data = yaml.load(master))

common = data['common']
for file_name in data:
    if file_name == 'common':
        continue
    data_out = ruamel.yaml.comments.CommentedMap()
    # you can leave out the following two lines of code if you do a safe_load()
    # but you will lose the ordering in your output file
    for k, v in common.items():
        data_out[k] = v
    for k, v in data[file_name].items():
        data_out[k] = v
    with open(file_name, 'wb') as fp:
        yaml.dump(data_out, stream=fp)

Given the following input in file master.file:

# YAML master file
common: &COMMON
  seo_title: general_title
  seo_description: seo_description
  seo_canonical: seo_canonical

product-conf.file:
  <<: *COMMON
  product_id: general_id
  title: general_title
  intro: general_intro

service-conf.file:
  <<: *COMMON
  service_id: general_id
  title: general_title
  products: list_products

running the program gives you two files, product-conf.file::

seo_title: general_title
seo_description: seo_description
seo_canonical: seo_canonical
product_id: general_id
title: general_title
intro: general_intro

and service-conf.file:

seo_title: general_title
seo_description: seo_description
seo_canonical: seo_canonical
service_id: general_id
title: general_title
products: list_products

It is also possible to have the three parts in separate input files instead of values in a combined file.

---

Alternatively if your real input files are not going to include anything that is handled in a special way by cpp you can do:

cpp -P -include master.in -o service-conf.file service-conf.in

with master.in:

seo_title: general_title
seo_description: seo_description
seo_canonical: seo_canonical

and service-conf.in:

service_id: general_id
title: general_title
products: list_products

this gives the same service-conf.out as with the previous example. product-conf.in would of course work in the same way.

The -P option for cpp suppresses debug output comments and -include includes the argument as if the first line of the input file has the preprocessor directive:

#include "master.in"

you can also make that explicit and leave out the commandline option.