I have a python script that outputs file with similar text:
1234\insert into\\default\e72303\FINISHED\False\23ms\N/A\37m10s\105\2017-
08-23 09:55:10.155407000\2017-08-23 09:55:10.178453000
This data is split by "\" and is imported to a table in hive database.
My issue that some of that data contains ^M carriage return character which splits up my data:
1234\INSERT INTO customer_touch.XXX_test_data_pickup^M
(^M
CI\default\e72303\FINISHED\False\331ms\0 / 0 ( 0%)\37m11s\0\2017-08-
23 09:55:08.066620000\2017-08-23 09:55:08.398299000
I need to remove ^M and have my data all together. I have tried dos2unix on the filename which does remove ^M but my data is still split.
Below is my code. I have crontab setup that outputs this into a text file
datanodes = ["https://XXXXXXX/",
"https://XXXXXXX"]
for i, datanode in enumerate(datanodes):
try:
response = requests.get(datanode + "queries?
json",auth=HTTPDigestAuth(XXX, XXX),verify='XXXX.pem')
data = response.json()
for query in data['completed_queries']:
print query['query_id'] + "\\" + query['stmt'][0:80] + "\\" + query['default_db'] + "\\" + query['effective_user'] + "\\" + query['state'] + "\\" + str(query['executing']) + "\\" + query['duration'] + "\\" + query['progress'] + "\\" + query['waiting_time'] + "\\" + str(query['rows_fetched']) + "\\" + query['start_time']+ "\\" + query['end_time']
except IOError as ioe:
print ioe
except Exception as e:
print(e)
I was able to remove ^M with replace('\r', '') per Charles Duffy's suggestion. I changed my code to query['stmt'][0:80].replace('\r', '')