Youtube data api asks for consent while accesstype is set to offline

Question

I have implemented the Google Youtube data api in a website, to let users see videos of certain channels. I have read the documentation from Google on how to use it, and used the offline accesstype since it lets the API autorize itself, refreshing the access token after it expires.

The API is working, but i still get prompt for autorization after 1 hour (default time for expiration of access token), while having the code that refresh the access token automatically.

I didnt provide all the code im using, only the code that handles the API setup and expiration of tokens, since the other code works fine.

Any help to what im doing wrong here would be greatly appreciated.
Updated

$client = new Google_Client();
$client->setAuthConfigFile($config);
$client->setRedirectUri($redirect);
$client->addScope('https://www.googleapis.com/auth/youtube.readonly');
$client->setAccessType('offline');

if(file_exits($aToken){
$accessToken = file_get_contents($aToken);
$client->setAccessToken($accessToken);

if($client->isAccessTokenExpired()) {
  if($client->getRefreshToken()){
    $client->fetchAccessTokenWithRefreshToken($client->getRefreshToken());
    $client->setAccessToken($client->getAccessToken());
    file_put_contents($aToken, $client->getAccessToken());
  } else {
      authorize();
    }
}

Answer 1

You are applying your refresh token but not actually using it to fetch a new access token.

$client->fetchAccessTokenWithRefreshToken($client->getRefreshToken());
$client->setAccessToken($client->getAccessToken()); 

My sample Oauth2Authentication.php