So, I want to search my whole directory for files that contain a list of regular expressions. That includes: directories, pdfs, and csv files. I can succesfully do this task when searching for only text files but search all file types is the struggle. Below is my work so far:
import glob
import re
import PyPDF2
#-------------------------------------------------Input----------------------------------------------------------------------------------------------
folder_path = "/home/"
file_pattern = "/*"
folder_contents = glob.glob(folder_path + file_pattern)
#Search for Emails
regex1= re.compile(r'\S+@\S+')
#Search for Phone Numbers
regex2 = re.compile(r'\d\d\d[-]\d\d\d[-]\d\d\d\d')
#Search for Locations
regex3 =re.compile("([A-Z]\w+), ([A-Z]{2})")
for file in folder_contents:
read_file = open(file, 'rt').read()
if readile_file == pdf:
pdfFileObj = open('pdf.pdf', 'rb')
pdfReader = PyPDF2.PdfFileReader(pdfFileObj)
pageObj = pdfReader.getPage(0)
content= pageObj.extractText())
if regex1.findall(read_file) or regex2.findall(read_file) or regex3.findall(read_file):
print ("YES, This file containts PHI")
print(file)
else:
print("No, This file DOES NOT contain PHI")
print(file)
When I run this i get this error:
YES, This file containts PHI
/home/e136320/sample.txt
No, This file DOES NOT contain PHI
/home/e136320/medicalSample.txt
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-129-be0b68229c20> in <module>()
19
20 for file in folder_contents:
---> 21 read_file = open(file, 'rt').read()
22 if readile_file == pdf:
23 # creating a pdf file object
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-128-1537605cf636> in <module>()
18
19 for file in folder_contents:
---> 20 read_file = open(file, 'rt').read()
21 if regex1.findall(read_file) or regex2.findall(read_file) or regex3.findall(read_file):
22 print ("YES, This file containts PHI")
/jupyterhub_env/lib/python3.5/codecs.py in decode(self, input, final)
319 # decode input (taking the buffer into account)
320 data = self.buffer + input
--> 321 (result, consumed) = self._buffer_decode(data, self.errors, final)
322 # keep undecoded input until the next call
323 self.buffer = data[consumed:]
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc7 in position 10: invalid continuation byte
Any suggestions?
You can't open a pdf file like that, it is expecting a plain text file. You could use something like this:
fn, ext = os.path.splitext(file)
if ext == '.pdf':
open_function = PyPDF2.PdfFileReader
else: # plain text
open_function = open
with open_function(file, 'rt') as open_file:
# Do something with open file...
This snippet checks the file extension then assigns an open function depending on what it finds, this is a bit naive and could be done better in a method similar to the one shown in this answer.