How can I remove the "\n" character in a large file with Python?

Question

I have a large .txt file containing numerous email addresses, but it also contains many unnecessary "\n" characters. I want to extract only the email addresses and remove any other characters.

To accomplish this, I have written a small script in Python.

import re

filename = "input.txt"
output_filename = "output.txt"
email_regex = r'\s*([A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,})\s*'

with open(filename, "r") as f, open(output_filename, "w") as out:
    for line in f:
        emails = re.findall(email_regex, line)
        for email in emails:
            out.write(email + "\n")

While the script successfully extracted regular email addresses, it encountered some difficulties with certain formats.

As an example, suppose I have a line of data that reads "CC\[email protected]\n". When I run my code, the resulting output is "[email protected]", which is not what I intended. Rather, I would like the output to be "[email protected]" without the leading "n" character."

Next, I tested another small Python script for a single email address, and the results were successful.

import re

string = "CC\[email protected]\n"
email_regex = r'[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}'

email = re.search(email_regex, string).group()

print(email)

So I want to get same result from a large file. If you have a solution for this, it would be good for me.

Answer 1

A shot in the dark: because you said that matching CC\[email protected]\n results in [email protected], I'll guess that your "\n characters" are not line breaks, but actual \s followed by ns. This can happen if somewhere in the pipeline the contents were incorrectly escaped, or if the text came from source code.

That would explain why your small example works with a hardcoded string, but not the text file: when you write string = "CC\[email protected]\n", Python itself is replacing the \n characters with a linebreak. To simulate the contents of your text file, you should instead use string = "CC\\[email protected]\\n".

If that's the case, you can either add a negative lookbehind to your regex, like (?<\\n)rest_of_email_regex_here.

Or more simply, do a preprocessing step of replacing all those characters with actual line breaks:

...
    for line in f:
        line = line.replace('\\n', '\n')
        emails = re.findall(email_regex, line)
        ...

If your text file really only contains \n and email addresses, after replacing the \n you can skip the regex and use line.split() to extract all addresses. This will return all sequences of non-space characters.