How to record http requests with Google Chrome extension and persist them

Question

I want to create a Chrome extension, that records HTTP requests (to a pre-defined host) and persists them as a list in local storage so when I call a particular website again the list will be extended.

I want to go with Manifest v3 to make the extension "ready for the future". I created a background script to trigger the request that currently puts all the details into local storage like that (currently this is redundant for demonstration purposes, I also tried it seperated):

chrome.webRequest.onBeforeRequest.addListener(details => {

    var urls = [];

    chrome.storage.local.get(['data'], function(data){

        urls = data.urls;

    });

    chrome.scripting.executeScript(
        {
          target: {tabId: details.tabId},
          func: recordClick,
          args: [details, urls]
        },
        () => { 
            urls.push(details); 
            console.log(urls.length);
            chrome.storage.local.set({urls: urls});
        });
    }, {
    urls: ['<all_urls>']

});

There's another function called recordClick() that does the same as in the callback:

function recordClick(details, urls) {

    urls.push(details.url);

    chrome.storage.local.set({urls: urls});

}

I tried several ways on where to load and save the result but none of them work. When I load the previous urls within the onBeforeRequest trigger, urls is not global and not known within the callback. When I put it outside the trigger definition, it's not reading the storage in realtime. I also tried to load the urls in a content script, loaded at "Document start". I tried to load the urls in the backend script at the top, and so on.

Seems like I have a timing problem: The trigger always loads an empty list or the variable is not global. I'm not able to extend the list. No matter where I put the storage functions.

Is my plan feasable at all? What am I'm doing wrong?

thanks!

Answer 1

Since chrome.storage.local.get is asynchronous, you should move chrome.scripting.executeScript into the callback of it.

onComplete may be suitable for your purpose, instead of onBeforeRequest.

chrome.webRequest.onBeforeRequest.addListener(details => {
    chrome.storage.local.get('urls', function(data){ 
        let urls = [];
        if( data.urls ) {
            urls = data.urls;
        }
        urls.push(details);
        chrome.storage.local.set({urls: urls}, function() {
            console.log('Value is set to ');
            console.log(urls);
        });

        chrome.scripting.executeScript( { 
             target: {tabId: details.tabId}, 
             func: function(details, urls){ console.log("executed script") }, 
             args: [details, urls] 
        }, 
        () => { 
            console.log("injected")
        }); 
    }); 
}, 
{ urls: ['<all_urls>'] }
);