I'm trying to scrape a website & put the value in cache so I don't hit the daily limit of UrlFetchApp
Here is the script I did:
/**
* Scrape URL, return whatever you choose with jquery-style selectors.
Dependency: cheeriogs, see https://github.com/fgborges/cheeriogs
*
* @param {url} valid start-url
* @return result (array values)
*
* @customfunction
*/
function scrapercache(url) {
var result = [];
var description;
var options = {
'muteHttpExceptions': true,
'followRedirects': false,
};
var cache = CacheService.getScriptCache();
var properties = PropertiesService.getScriptProperties();
try {
let res = cache.get(url);
if (!res) {
// trim url to prevent (rare) errors
url.toString().trim();
var r = UrlFetchApp.fetch(url, options);
var c = r.getResponseCode();
// check for meta refresh if 200 ok
if (c == 200) {
var html = r.getContentText();
cache.put(url, "cached", 21600);
properties.setProperty(url, html);
var $ = Cheerio.load(html); // make sure this lib is added to your project!
// meta description
if ($('meta[name=description]').attr("content")) {
description = $('meta[name=description]').attr("content").trim();
}
}
result.push([description]);
}
}
catch (error) {
result.push(error.toString());
}
finally {
return result;
}
}
but when I call the function like that:
=scrapercache("https://www.gurufocus.com/term/total_freecashflow/nyse:ABBV/Free-Cash-Flow")
I get the error message:
Exception: argument too large: value
Anyone can help me please? Thank you :) Gabriel
As written in the official documentation,
The maximum length of a key is 250 characters. The maximum amount of data that can be stored per key is 100KB.
If the size of the data put in cache exceeds any of the above limitations, the error
Exception: argument too large
is shown. In your case, value
exceeds 100KB. Solution would be to cache only necessary data or don't cache at all depending on your specific needs.