How to Programmatically communicate with Restful API using SSH tunnelling

I am building a Restful API client using .NET core framework. The APIs are OpenStack API, however, because of the network configuration, I cannot access the API from my local computer (also development computer), I have to ssh into a machine that can ssh into the OpenStack infrastructure when accessing OpenStack normally.

Bearing this in mind, is it possible to use SSH tunnel for the API endpoints and then call it in the implemented Web API client? I have tried to do this, but the call to the endpoint returns error 401 - content length is required.

Solution

Basically its possible to call the Openstack API endpoint through an SSH-tunnel without any public accessible API-endpoints. Bacause I have no experience with .NET core framework this answer is really generic without C# code. I hope it helps you anyway.

IMPORTANT: You can use the following steps only, when you have a admin-login to the openstack-deployment and you should ONLY!!! use this way, when the openstack-deployment is a test-deployment, where breaking the deployment doesn't affect other users.

1. SSH-tunnel

You can forward ports with the command:

ssh -L 127.0.0.1:<PORT>:<IP_REMOTE>:<PORT> <USER_JUMPHOST>@<IP_JUMPHOST> -fN

<PORT> = Port of the Openstack-Component you want to access remotely (for example 5000 for keystone)
<IP_REMOTE> = IP of the host, where your openstack deployment is running
<USER_JUMPHOST>@<IP_JUMPHOST> = ssh-access to the jumphost, which is between you and your openstack deployment

This has to be done for each openstack component. If you don't want this command in the backgroup remove the -fN at the end.

Here at first you have to forward Keystone with port 5000.

example: ssh -L 127.0.0.1:5000:192.168.62.1:5000 deployer@192.168.67.1 -fN

You can test the access via curl or webbrowser from your local pc:

curl http://127.0.0.1:5000
{"versions": {"values": [{"id": "v3.13", "status": "stable", "updated": "2019-07-19T00:00:00Z", "links": [{"rel": "self", "href": "http://127.0.0.1:5000/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}

2. change openstack-endpoints

To be also able to login on the openstack deployment through the tunnel, you have to change the endpoints to listen to the localhost on the remote system too, where your openstack depoyment is:

Login normally on your openstack deployment as admin-user.
List all endpoints: openstack endpoint list
Change the public and internal-endpoint of keystone to localhost:

openstack endpoint set --url http://127.0.0.1:5000 <ID_OF_INTERNAL_KEYSTONE_ENDPOINT>

After changing the internal endpoint it will break the openstack-login on the remotesystem for now, but don't worry.

Now you can login to the openstack via openstack-client from your local pc. Here you have to authorize against local-host. If you use an rc-file to login, you have to change the auth-url to export OS_AUTH_URL=http://127.0.0.1:5000/v3
Change the nova endpoints by running on your local pc openstack endpoint set --url "http://127.0.0.1:8774/v2.1" <ID> for the internal and public endpoint of nova to run commands like openstack server list through your ssh-tunnel (of course you need also an ssh-tunnel for port 8774) to do this.

3. Authorize against the openstack-deployment

When you send HTTP-Requests without the openstack-client, you have to manually request an authentication token from the deployment:

Login normally on your openstack deployment
Make a Token-Request:

curl -v -s -X POST "$OS_AUTH_URL/auth/tokens?nocatalog" -H "Content-Type: application/json" -d '{ "auth": { "identity": { "methods": ["password"],"password": {"user": {"domain": {"name": "'"$OS_USER_DOMAIN_NAME"'"},"name": "'"$OS_USERNAME"'", "password": "'"$OS_PASSWORD"'"} } }, "scope": { "project": { "domain": { "name": "'"$OS_PROJECT_DOMAIN_NAME"'" }, "name": "'"$OS_PROJECT_NAME"'" } } }}' --stderr - | grep X-Subject-Token

This command can be used without changes. The Value after the Key X-Subject-Token is the token from Keystone. Copy this value and export the token as the environment variable OS_TOKEN. For example like the following line

export OS_TOKEN=gAAAAABZuj0GZ6g05tKJ0hvihAKXNJgzfoT4TSCgR7cgWaKvIvbD66StJK6cS3FqzR2DosmqofnR_N-HztJXUcVhwF04HQsY9CBqQC7pblGnNIDWCXxnJiCH_jc4W-uMPNA6FBK9TT27vE5q5AIa487GcLLkeJxdchXiDJvw6wHty680eJx3kL4

Make requests with the token.

For example GET-Requests with curl:

curl -s -X GET -H "X-Auth-Token: $OS_TOKEN" http://127.0.0.1:5000/v3/users | python -m json.tool