How to remove multiline text from matching strings?

I have the following snippet I would like to completely remove with a regex (or some other method).

# ---------------------------------------------------------------------------------------------------------------------
# MODULE PARAMETERS
# These are the variables we have to pass in to use the module specified in the terragrunt configuration above
# ---------------------------------------------------------------------------------------------------------------------

Is there syntax to tell the match to get rid of everything between the two matches?

It seems like it should be easy to do, but for some reason I have only been able to find a regex that pulls the first and last match out via this code. I have tried a number of permutations of this regex but can't get it working.

...
re.sub(r'# --*[\S\s]*---', '', lines[line])
...

This regex tool says that my regex should work.

EDIT:

The text I am interested in matching is being read in one line at a time.

...
for the_file in files_to_update:
    with open(the_file + "/the_file", "r") as in_file:
        lines = in_file.readlines()

And subsequently being iterated over. The snippet above is acutally happening in this loop.

for line in range(len(lines)):

Solution

You should read in the file into a single variable to be able to run a regex on it, that can match more than one line of text.

You may use

with open(filepath, 'r') as fr:
  with open(filesavepath, 'w') as fw:
    fw.write( re.sub(r'^# -+(?:\n# .*)*\n# -+$\n?', '', fr.read(), flags=re.M) )

See the Python demo and a regex demo.

Here, fr is the handle of the file you read from, and fw is the handle of the file you are writing to. The input for re.sub is fr.read(), this method grabs the whole file contents and passes to the regex engine.

The regex means:

^ - start of a line (due to re.M)
# -+ - a #, space and then one or more hyphens
(?:\n# .*)* - 0 or more repetitions of a newline, #, space, any text up to the end of a line
\n - a newline
# -+$ - #, space, one or more hyphens and end of a line
\n? - an optional newline.

A non-regex way of removing the comments is by reading line by line, checking if a line starts with # --- and set a flag that would enable checking if we are inside the comment or not:

for line in fr:
    if line.startswith('# ---'):
        flag = not flag
        continue
    if flag:
        lines.append(line)
        
print("\n".join(lines))

See this Python demo.