Scenario
I am creating a nodejs server, which'll act as a middle server between the actual client and actual server. i.e. I send a request to a website, through my nodejs server, receive the response from actual (website) server, and forward the same to the client (the browser).
Here's part of the code for doing that
const cheerio = require('cheerio');
//#================================================================
// include other files and declare variables
//#================================================================
app.get('/*', (req, res) => {
//#================================================================
// some code...
//#================================================================
request(options, function(error, response, body){
if (!error && response.statusCode == 200) {
res.writeHead(200, headers);
if (String(response.headers['content-type']).indexOf('text/html') !== -1){
var $ = cheerio.load(body);
//#================================================
// perform html manipulations
//#================================================
//send the html content as response
res.end($.html());
}else{
res.end(body);
}
}else{
res.send({status: 500, error: error});
}
});
}
Everything works fine, untill I stumble upon this particular website https://www.voonik.com/recommendations/bright-cotton-a-line-kurta-for-women-blue-printed-bcown-007b-38-1f2073ca.
If you look at its view source it is more or less like this
<!doctype html>
<html lang="en-in" data-reactid=".mc12nbyapk" data-react-checksum="-2121099716">
<!-- rest of the html code -->
...
<script type="text/javascript" charset="UTF-8" data-reactid=".mc12nbyapk.1.1">
window.NREUM||(NREUM={});NREUM.info = {"agent":"","beacon":"bam.nr-data.net","errorBeacon":"bam.nr-data.net"...
...
</script></body></html>
and when I send this very html in my response object, it sends incomplete html i.e. breaks in between somewhere of the last script tag.
I consoled log the html also and it prints the whole string. But sending the same in response object sends half.
Also tried res.write(); res.send() and storing the html content in a variable then sending that variable, but the outcome is same i.e. incomplete html content.
I was thinking of solution which wouldn't involve writing to and reading from a file. Just directly send the response as you receive it
after you manipulate the target server response contents, the content length is changed, so you must recalculate the content length and rewrite the content-length header, or just delete the content-length header,
put this code delete headers['content-length'], before res.writeHead(200, headers); this line.