Ocelot api gateway - kubernetes - error: "namespace:serviceservice:managementservice Unable to use ,it is invalid. Address must contain host only...."

The problem that i am facing is that the ocelot kubernetservicediscorverProvider does not seem to find the other services on the name space in kubernetes.My goal is to use api gateway to call apis in the other services in the same namespace. I currently get a http 404 Not Found error. And the api gateway pod, logs the following:

Ocelot.Provider.Kubernetes.KubernetesServiceDiscoveryProvider[0] requestId: 0HM93C93DL2T0:00000003, previousRequestId: no previous request id, message: namespace:serviceservice:managementservice Unable to use ,it is invalid. Address must contain host only e.g. localhost and port must be greater than 0

warn: Ocelot.Responder.Middleware.ResponderMiddleware[0] requestId: 0HM93C93DL2T0:00000003, previousRequestId: no previous request id, message: Error Code: ServicesAreEmptyError Message: There were no services in NoLoadBalancer errors found in ResponderMiddleware. Setting error response for request path:/api/management/User/3910, request method: GET

I suspect that i have mis configured something. I first tried using the Ocelot documentation, regarding kubernetes, but the documentation is out dated. (an example is the Type the sugest value does not work for more info go this github issue Docs/Kubernetes provider are wrong)

Then i went on searching online through github issues, stack overflow posts and even the source code. But i do not see have what i am lacking in my config.

I currently have kubernetes running localy, with minikube. The only things that i have seen online is that others have misconfigured the ocelot.json. But i do not see what i have done incorrectly in my config.

(Before trying ocelot on kubernetes i first try it with local hosts, to try out if it works and to see what it lacks. It apparantly lacked a middleware that could control jwt with different roles which had right to acces the end point. I have now written the middleware my self and it works on the local host config for ocelot)

My ocelot.json config file looks like this for kubernetes:

{
  "Routes": [
    {
      "UpstreamPathTemplate": "/api/management/User/{everything}",
      "UpstreamHttpMethod": [ "POST", "PUT", "GET" ],
      "DownstreamPathTemplate": "/api/management/User/{everything}",
      "DownstreamScheme": "http",
      "ServiceName": "managementservice",
      "AuthenticationOptions": {
        "AuthenticationProviderKey": "Bearer",
        "AllowedScopes": [ "CompanyId", "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier" ]
      },
      "RouteClaimsRequirement": { "role": "1,2,3" },
      "AddHeadersToRequest": {
        "CompanyId": "Claims[CompanyId] > value",
        "UserId": "Claims[http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier] > value"
      }
    }
  ],
  "GlobalConfiguration": {
    "ServiceDiscoveryProvider": {
      "Host": "127.0.0.1",
      "Port": 8083,
      "Namespace": "service",
      "Type": "KubernetesServiceDiscoveryProvider"
    }
  }
}

my startup.cs ConfigureServices method looks like this

public void ConfigureServices(IServiceCollection services)
        {
            services.AddCors(c =>
            {
                c.AddPolicy("AllowOrigin", options => options.WithOrigins(Configuration["Cors:AllowOrigins"])
                    .AllowAnyHeader()
                    .AllowAnyMethod().AllowCredentials());
            });

            #region Authication settings

            TokenValidationParameters tokenValidationParameters = new TokenValidationParameters
            {
                ValidateIssuerSigningKey = true,
                IssuerSigningKey = new SymmetricSecurityKey(Encoding.ASCII.GetBytes(Configuration["Jwt:Key"])),
                ValidateIssuer = false,
                ValidateAudience = false,
                ValidateLifetime = true,
                ClockSkew = TimeSpan.Zero
            };

            services.AddSingleton(tokenValidationParameters);
            services.AddAuthentication(
                    x =>
                    {
                        x.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
                        x.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme;
                    }
                )
                .AddJwtBearer(JwtBearerDefaults.AuthenticationScheme, x =>
                {

                    x.RequireHttpsMetadata = false;
                    x.SaveToken = true;
                    x.TokenValidationParameters = tokenValidationParameters;
                });

            #endregion

       //Some more code

       services.AddOcelot().AddKubernetes();
       }

my startup.cs Configure method looks like this

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            // some more code

            app.UseOcelot(configuration);
        }

my program.cs CreateHostBuilder method looks like this

public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
               .ConfigureAppConfiguration((hostingContext, config) =>
               {
                   config.AddJsonFile("secrets/appsettings.kubernetes.json", optional: true)
                         .AddJsonFile("ocelot.json");
               })
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                });

Solution

It turns out that the real problem that i was having was in the permissions in kubernetes. The ocelot documentation also mentions this. But the command in the documentation is incorrect (most likely out dated).

This is the command that i used. Be warned the kubernetes documenation strongly disrecommend usage of this command permissive-rbac-permissions. But it is at least a way for you to test your api gateway in ocelot locally.

kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts