I am trying to install a Jupyterhub on GKE cluster
By following steps mentioned in below
github.com GoogleCloudPlatform/ai-notebooks-extended main/gke-hub-example
JupyterHub got sucesfully installed, but when i try to start a Notebook instance , it throws the below error
Error: HTTP 500: Internal Server Error (Error in Authenticator.pre_spawn_start: ValueError No such profile: [‘gcr-io-noted-amphora-248618-jupyter-mine-basic’]. Options include: gcr-io-noted-amphora-248618-jupyter-mine-basic, gcr-io-noted-amphora-248618-jupyter-mine-basic)
Below are the log messages
2021-06-02T10:06:39.620382674Z Watches established. E
2021-06-02T10:06:41.466552969Z [I 2021-06-02 10:06:41.466 JupyterHub app:2332] Running JupyterHub version 1.2.2 E
2021-06-02T10:06:41.466680449Z [I 2021-06-02 10:06:41.466 JupyterHub app:2362] Using Authenticator: gcpproxiesauthenticator.gcpproxiesauthenticator.GCPProxiesAuthenticator E
2021-06-02T10:06:41.466780251Z [I 2021-06-02 10:06:41.466 JupyterHub app:2362] Using Spawner: kubespawner.spawner.KubeSpawner-1.0.0 E
2021-06-02T10:06:41.466891898Z [I 2021-06-02 10:06:41.466 JupyterHub app:2362] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.2.2 E
2021-06-02T10:06:41.469256528Z [I 2021-06-02 10:06:41.469 JupyterHub app:1457] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret E
2021-06-02T10:06:41.494694057Z [I 2021-06-02 10:06:41.494 alembic.runtime.migration migration:155] Context impl SQLiteImpl. E
2021-06-02T10:06:41.495207743Z [I 2021-06-02 10:06:41.494 alembic.runtime.migration migration:158] Will assume non-transactional DDL. E
2021-06-02T10:06:41.520272984Z [I 2021-06-02 10:06:41.519 alembic.runtime.migration migration:517] Running stamp_revision → 4dc2d5a8c53c E
2021-06-02T10:06:41.696420888Z [I 2021-06-02 10:06:41.690 JupyterHub proxy:461] Generating new CONFIGPROXY_AUTH_TOKEN E
2021-06-02T10:06:41.696456030Z [W 2021-06-02 10:06:41.692 JupyterHub app:1687] No admin users, admin interface will be unavailable. E
2021-06-02T10:06:41.696467162Z [W 2021-06-02 10:06:41.692 JupyterHub app:1688] Add any administrative users to c.Authenticator.admin_users in config. E
2021-06-02T10:06:41.696476364Z [I 2021-06-02 10:06:41.692 JupyterHub app:1717] Not using allowed_users. Any authenticated user will be allowed. E
2021-06-02T10:06:41.764431054Z [I 2021-06-02 10:06:41.760 JupyterHub app:2399] Initialized 0 spawners in 0.005 seconds E
2021-06-02T10:06:41.766431671Z [W 2021-06-02 10:06:41.765 JupyterHub proxy:655] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else… E
2021-06-02T10:06:41.766459045Z [I 2021-06-02 10:06:41.765 JupyterHub proxy:659] Starting proxy @ http://:8080/ E
2021-06-02T10:06:42.795514818Z 10:06:42.790 [ConfigProxy] e[32minfoe[39m: Proxying http://*:8080 to (no default) I
2021-06-02T10:06:42.795544011Z 10:06:42.794 [ConfigProxy] e[32minfoe[39m: Proxy API at http://127.0.0.1:8005/api/routes I
2021-06-02T10:06:42.848453299Z 10:06:42.847 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes I
2021-06-02T10:06:42.849027936Z [I 2021-06-02 10:06:42.848 JupyterHub app:2647] Hub API listening on http://0.0.0.0:8081/hub/ E
2021-06-02T10:06:42.849067692Z [I 2021-06-02 10:06:42.848 JupyterHub app:2649] Private Hub API connect url http://10.56.0.13:8081/hub/ E
2021-06-02T10:06:42.852807313Z 10:06:42.852 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes I
2021-06-02T10:06:42.853384703Z [I 2021-06-02 10:06:42.853 JupyterHub proxy:320] Checking routes E
2021-06-02T10:06:42.853536476Z [I 2021-06-02 10:06:42.853 JupyterHub proxy:400] Adding default route for Hub: / => http://10.56.0.13:8081 E
2021-06-02T10:06:42.857050192Z 10:06:42.856 [ConfigProxy] e[32minfoe[39m: Adding route / → http://10.56.0.13:8081 I
2021-06-02T10:06:42.857766510Z 10:06:42.857 [ConfigProxy] e[32minfoe[39m: Route added / → http://10.56.0.13:8081 I
2021-06-02T10:06:42.858271548Z 10:06:42.858 [ConfigProxy] e[32minfoe[39m: 201 POST /api/routes/ I
2021-06-02T10:06:42.860752421Z [I 2021-06-02 10:06:42.859 JupyterHub app:2722] JupyterHub is now running at http://:8080/ E
2021-06-02T10:11:36.698005265Z [I 2021-06-02 10:11:36.697 JupyterHub log:181] 302 GET / → /hub/ (@::ffff:10.56.0.14) 1.24ms E
2021-06-02T10:11:36.961851801Z [I 2021-06-02 10:11:36.961 JupyterHub log:181] 302 GET /hub/ → /hub/login?next=%2Fhub%2F (@::ffff:10.56.0.14) 0.84ms E
2021-06-02T10:11:37.224288918Z [I 2021-06-02 10:11:37.223 JupyterHub gcpproxiesauthenticator:64] user_email is phani111@gmail.com E
2021-06-02T10:11:37.247973936Z [I 2021-06-02 10:11:37.247 JupyterHub gcpproxiesauthenticator:109] username is phani111@gmail.com E
2021-06-02T10:11:37.248073429Z [I 2021-06-02 10:11:37.247 JupyterHub gcpproxiesauthenticator:110] user.name is phani111@gmail.com E
2021-06-02T10:11:37.306125736Z [I 2021-06-02 10:11:37.305 JupyterHub reflector:271] watching for pods with label selector=‘component=singleuser-server’ in namespace default E
2021-06-02T10:11:37.319422003Z [I 2021-06-02 10:11:37.319 JupyterHub reflector:271] watching for events with field selector=‘involvedObject.kind=Pod’ in namespace default E
2021-06-02T10:11:37.326054796Z [I 2021-06-02 10:11:37.325 JupyterHub log:181] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:10.56.0.14) 102.76ms E
2021-06-02T10:11:38.661888708Z [I 2021-06-02 10:11:38.661 JupyterHub log:181] 302 GET /hub/ → /hub/spawn (phani111@gmail.com@::ffff:10.56.0.14) 10.97ms E
2021-06-02T10:11:38.922734475Z [I 2021-06-02 10:11:38.922 JupyterHub log:181] 200 GET /hub/spawn (phani111@gmail.com@::ffff:10.56.0.14) 10.46ms E
2021-06-02T10:11:42.863160490Z 10:11:42.862 [ConfigProxy] e[32minfoe[39m: 200 GET /api/routes I
2021-06-02T10:11:42.864090929Z [I 2021-06-02 10:11:42.863 JupyterHub proxy:320] Checking routes E
2021-06-02T10:11:43.150114686Z [E 2021-06-02 10:11:43.149 JupyterHub user:690] Unhandled error starting phani111@gmail.com’s server: No such profile: [‘gcr-io-noted-amphora-248618-jupyter-mine-basic’]. Options include: gcr-io-noted-amphora-248618-jupyter-mine-basic, gcr-io-noted-amphora-248618-jupyter-mine-basic E
2021-06-02T10:11:43.192499074Z [E 2021-06-02 10:11:43.191 JupyterHub pages:261] Failed to spawn single-user server with form E
2021-06-02T10:11:43.192554817Z Traceback (most recent call last): E
2021-06-02T10:11:43.192568481Z File “/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/pages.py”, line 257, in post E
2021-06-02T10:11:43.192577262Z return await self._wrap_spawn_single_user( E
2021-06-02T10:11:43.192588011Z File “/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/pages.py”, line 314, in _wrap_spawn_single_user E
2021-06-02T10:11:43.192596671Z raise web.HTTPError( E
2021-06-02T10:11:43.192603972Z tornado.web.HTTPError: HTTP 500: Internal Server Error (Error in Authenticator.pre_spawn_start: ValueError No such profile: [‘gcr-io-noted-amphora-248618-jupyter-mine-basic’]. Options include: gcr-io-noted-amphora-248618-jupyter-mine-basic, gcr-io-noted-amphora-248618-jupyter-mine-basic) E
2021-06-02T10:11:43.192613428Z E
2021-06-02T10:11:43.193734621Z [I 2021-06-02 10:11:43.193 JupyterHub log:181] 200 POST /hub/spawn (phani111@gmail.com@::ffff:10.56.0.14) 96.59ms E
2021-06-02T10:11:48.350476518Z [E 2021-06-02 10:11:48.347 JupyterHub user:690] Unhandled error starting phani111@gmail.com’s server: No such profile: [‘gcr-io-noted-amphora-248618-jupyter-mine-basic’]. Options include: gcr-io-noted-amphora-248618-jupyter-mine-basic, gcr-io-noted-amphora-248618-jupyter-mine-basic E
2021-06-02T10:11:48.373694792Z [E 2021-06-02 10:11:48.373 JupyterHub pages:261] Failed to spawn single-user server with form E
2021-06-02T10:11:48.373797482Z Traceback (most recent call last): E
2021-06-02T10:11:48.373814126Z File “/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/pages.py”, line 257, in post E
2021-06-02T10:11:48.373821806Z return await self._wrap_spawn_single_user( E
2021-06-02T10:11:48.373829752Z File “/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/pages.py”, line 314, in _wrap_spawn_single_user E
2021-06-02T10:11:48.373837262Z raise web.HTTPError( E
2021-06-02T10:11:48.373844504Z tornado.web.HTTPError: HTTP 500: Internal Server Error (Error in Authenticator.pre_spawn_start: ValueError No such profile: [‘gcr-io-noted-amphora-248618-jupyter-mine-basic’]. Options include: gcr-io-noted-amphora-248618-jupyter-mine-basic, gcr-io-noted-amphora-248618-jupyter-mine-basic) E
2021-06-02T10:11:48.373866321Z E
2021-06-02T10:11:48.375421105Z [I 2021-06-02 10:11:48.374 JupyterHub log:181] 200 POST /hub/spawn (phani111@gmail.com@::ffff:10.56.0.14) 76.97ms E
please do let me know how to debug this further.
It seems related to Kubespawner 1.0.0 version. When set to the previous one 0.16.1, it seems to work. Please try pulling the latest code from ai-notebook-extended where kubespawner version is set as a quick fix.
See: https://github.com/GoogleCloudPlatform/ai-notebooks-extended/issues/12