I use the below code to remove all HTML tags from a file and convert it to a plain text. Moreover, I have to convert XML/HTML characters to ASCII ones. Here, I have 21 lines which read whole the text. It means if I want to convert a huge file, I have to expend a lot of resource to do this.
Do you have any idea to increase the efficiency of the code and increase its speed while decrease the usage of the resources?
# -*- coding: utf-8 -*-
import re
# This file contains HTML.
file = open('input-file.html', 'r')
temp = file.read()
# Replace Some XML/HTML characters to ASCII ones.
temp = temp.replace ('‘',"""'""")
temp = temp.replace ('’',"""'""")
temp = temp.replace ('“',"""\"""")
temp = temp.replace ('”',"""\"""")
temp = temp.replace ('‚',""",""")
temp = temp.replace ('′',"""'""")
temp = temp.replace ('″',"""\"""")
temp = temp.replace ('«',"""«""")
temp = temp.replace ('»',"""»""")
temp = temp.replace ('‹',"""‹""")
temp = temp.replace ('›',"""›""")
temp = temp.replace ('&',"""&""")
temp = temp.replace ('–',""" – """)
temp = temp.replace ('—',""" — """)
temp = temp.replace ('®',"""®""")
temp = temp.replace ('©',"""©""")
temp = temp.replace ('™',"""™""")
temp = temp.replace ('¶',"""¶""")
temp = temp.replace ('•',"""•""")
temp = temp.replace ('·',"""·""")
# Replace HTML tags with an empty string.
result = re.sub("<.*?>", "", temp)
print(result)
# Write the result to a new file.
file = open("output-file.txt", "w")
file.write(result)
file.close()
The problem of using sting.tranlate()
or string.maketran()
is that when I use them I have to assign A char to another one. e.g.
print string.maketran("abc","123")
But, I need to assign a HTML/XML char like ‘
to the single quotation ('
) in ASCII. It means that I have to use the following code:
print string.maketran("‘","'")
It faces the following error:
ValueError: maketrans arguments must have same length
Whereas, if I use HTMLParser, it will convert all HTML/XML to ASCII without the above problem. I also have added a encode('utf-8')
to solve the following error:
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 246: ordinal not in range(128)
# -*- coding: utf-8 -*-
import re
from HTMLParser import HTMLParser
# This file contains HTML.
file = open('input-file.txt', 'r')
temp = file.read()
# Replace all XML/HTML characters to ASCII ones.
temp = HTMLParser.unescape.__func__(HTMLParser, temp)
# Replace HTML tags with an empty string.
result = re.sub("<.*?>", "", temp)
# Encode the text to UTF-8 for preventing some errors.
result = result.encode('utf-8')
print(result)
# Write the result to a new file.
file = open("output-file.txt", "w")
file.write(result)
file.close()