I've been banging my head against the wall at this for going on 16 hours and I can't seem to find anybody with the exact same issue.
I have a command line program (I'll just call it the CLI) I'm using that takes as an argument a text file, where each line of the file has important info. Without going into detail, the CLI passes the lines of the file to a web service where some operations are done.
I have a python program I'm writing where I create the file that gets passed to the CLI programmatically (using subprocess). The issue seems to be that the webservice is receiving a carriage return somewhere which is doesn't know how to handle. I was pretty careful to avoid any carriage returns in my creation of each of the lines that go into that file, but clearly carriage returns were getting added to the end. I opened my file in binary mode and sure enough, there was the carriage return. Bummer.
No matter, I see a script shared around here all the time showing how to convert a file from CRLF to LF. Added those handful of lines to my python script, and double checked and sure enough, when reading the file in binary mode, the carriage return was gone. hurray.
But when I go to run the script again, for some reason, the web service is still receiving carriage returns.
I suspect it has something to do with the python temporary file system I'm using. Maybe it works some weird way under the hood and after i'm changing the file to use LF and printing its contents, at some point its changing it back (maybe when its passed as an argument to the CLI)
I guess my question is, either 1) Can you think of something tempfile might be doing to mess with my file endings even after it looks like I had changed them? Maybe when the temp file gets passed into the cli with subprocess its reverting somehow? I'm just sending it with temp.name.
Or 2) Maybe I can do something other than python's built in tempfile. Its really convenient, so if I were going to not use it, are there some best practices for creating and deleting temporary files manually I should know about? Is there a best place to create them? For example, is it considered bad practice to create and then delete files right off the current directory?
Anyway, here's the relevant part of my code AFTER the changes to the line endings:
def my_function(mylist, my_id, local_id, args):
temp = tempfile.NamedTemporaryFile(mode='w+b', delete=False)
for each in mylist:
is_directory = False
destination = args.destination.replace("\\", "/")
if each["smallpath"] == "/":
full_path = each["smallpath"] + "/"
else:
full_path = each["rel_path"].replace("\\", "/") + "/" + each["specific_path"].lstrip("/")
if os.path.basename(full_path) == "":
is_directory = True
if is_directory is False:
line = f'"{full_path}" "~/{destination}/{each["orgID"]}-{each["userID"]}/{os.path.basename(full_path)}" \n'
else:
if each["smallpath"] != "/":
slash_index = full_path.rstrip('/').rfind("/")
local_dir = full_path[slash_index:].rstrip().rstrip('/')
else:
local_dir = "/"
line = f'"{full_path}" "~/{destination}/{each["orgID"]}-{each["userID"]}/{local_dir.lstrip("/")}" --recursive \n'
line = line.replace("\\", "/")
temp.write(line)
temp.seek(0)
windows_ending = b'\r'
unix_ending = b'\n'
double_newline = b'\n\n'
with open(temp.name, 'rb') as file:
file_contents = file.read()
temp.seek(0)
file_contents = file_contents.replace(windows_ending, unix_ending)
file_contents = file_contents.replace(double_newline, unix_ending)
with open(temp.name, 'wb') as file:
file.write(file_contents)
myprocess = subprocess.Popen(["programname", "commandname", myid, local_id, "--batch",
temp.name], stdout=subprocess.PIPE)
mycommand = myprocess.communicate()[0].decode('utf-8')
temp.close()
os.unlink(temp.name)
I wanted to thank everyone who tried to help. The comments helped me discover another vulnerability elsewhere in my code. The problem ended up actually not being my own, rather it was the service I was using