using CI_JOB_TOKEN for API call inside CI pipeline returns 404

Question

I have only these two jobs in my pipeline. The second one is supposed to retrieve a list of all previous jobs.

stages:
  - build
  - test

build:
  stage: build
  script:
   -  echo "build"

test_job:
  image: bash:latest
  stage: test
  script:
    - apk add curl
    - |
      curl --header "Job-Token: $CI_JOB_TOKEN" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/pipelines/${CI_PIPELINE_ID}/jobs"

My problem is that the second job returns a 404.

{
  "message": "404 Project Not Found"
}

This has to be because of authorization issues as the project defintely exists. However I am trying to access the project via API from itself and according to docs:

"By default, the allowlist of any project only includes itself."

Which means that CI_JOB_TOKEN should have sufficient access.

Further, which confuses me even more, when I replace "Job-Token: $CI_JOB_TOKEN" with "Private-Token: $CI_JOB_TOKEN", the curl command returns:

{
  "error": "invalid_token",
  "error_description": "Token was revoked. You have to re-authorize from the user."
}

What setings could be set incorrectly or am I using the API wrong?

Note: I tested the curl with a personal access token and it worked as expected

Answer 1

Job tokens have limited API access. You can't use this jobs API with a job token. Even when added to the allowlist, job tokens are still limited to specific endpoints, which does not include the jobs list API (though you can get job artifacts using a job token).

You will need to use another method to authenticate to this API, such as a project access token, group access token, or similar.