Creating a warc record with requests.get() response using warcio

Question

I'm using the warcio library to read and write warc files.

When trying to write a record of a response object from requests.get(URL,stream=False), warcio is writing only HTTP headers to the record but not the payload. However, when stream mode is enabled it works fine.

Is there a way store the payload when stream mode is not enabled?

Answer 1

I've found a workaround but not sure if it's the correct way. Instead of making request object streamable, I've made the payload streamable BytesIO(response.text.encode()) and this seems to work.