How would I replace all the sentences and paragraphs with a <string> tag in text files?
I want to keep spacing, tabs, and lists in the text document intact:
Example input:
Clause 1:
a) detail 1. some more about detail 1. Here is more information about this paragraph right here. There is more information that we think sometimes.
b) detail 2. some more about detail 2. and some more..
Example output:
<string>
a) <string>
b) <string>
I don't know if this is the best way, but it's fairly straightforward, and easy to modify. It handles the example in your problem statement, plus most of the example from your comment.
import sys, re
text = sys.stdin.read()
# A pattern expressing the parts of the input that we want to preserve:
keeper_pattern = r'''(?x) # verbose format
( # We put parens around the whole pattern
# (and use ?: for subgroups)
# so that when we use it as the splitter-pattern for re.split(),
# the result contains one string for each occurrence of the pattern
# (in addition to the usual between-splitter strings).
# The main thing we want to keep is paragraph-separators,
# and the 'lead' of the line that follows a para-sep:
#
\n{2,} # two or more newlines, followed by
\x20* # optional indentation (zero or more spaces), followed by
(?: # an optional item-marker, which is
(?: # either
\d+ \. # digits followed by a dot,
| # or
[a-z] \) # a letter followed by a right-paren,
) # followed by
\x20+ # one or more spaces.
)?
|
# The other thing we want to keep is
# item-markers within paragraphs:
#
\( i+ \) # a lower-case Roman numeral between parens
# (generalize as necessary)
)
'''
for (i, chunk) in enumerate(re.split(keeper_pattern, text)):
# In the result of re.split(),
# the splitters (keepers) will be in the odd positions.
is_keeper = (i % 2 == 1)
if is_keeper:
if chunk.startswith('\n'):
# paragraph-separator etc
replacement = chunk
else:
# within-para item-marker
replacement = ' ' + chunk + ' '
else:
if chunk == '':
# (happens if two keepers are adjacent)
replacement = ''
else:
# everything else
replacement = '<string>'
sys.stdout.write(replacement)