When I try to save the responses to a file, the actual response is not saved even though it shows in the console. The result that is saved in the file is None. See examples below
from concurrent.futures import ThreadPoolExecutor
import requests
#from timer import timer
######### create test file
URLsTest = '''
https://en.wikipedia.org/wiki/NBA
https://en.wikipedia.org/wiki/NFL
'''.strip()
with open('input.txt', 'w') as f:
f.write(URLsTest)
####################
with open('input.txt', 'r') as f:
urls=f.read().split('\n') # url list
def fetch(tt): # received tuple
session, url = tt
print('Processing')
with session.get(url) as response:
print(response.text)
#@timer(1, 5)
def main():
with ThreadPoolExecutor(max_workers=100) as executor:
with requests.Session() as session: # for now, just one session
results = executor.map(fetch, [(session, u) for u in urls]) # tuple list (session, url), each tuple passed to function
executor.shutdown(wait=True)
# write all results to text file
with open('output.txt', 'w') as f2:
for r in results: # tuple (url, html)
f2.write("%s\n" % r)
main()
Response file - output.txt
None
None
First of all, you could avoid printing the html since you are saving that output to a file. That way you can avoid using resources to print the results.
Then, your fetch is not returning anything for the results
. Therefore you should change your print
for a return
So instead of printing return the response.text
# print(response.text)
return response.text