After finding out that ASP.NET Core Module acts as a SSL terminator and thus always hosts on http, my question is how to propagate coming from a https address properly into IdentityServer. If you have set the headers X-Forwarded-Proto or X-Forwarded-Schema with the value https, should this not be picked up? From what I could tell from your BaseUrlMiddleware.cs code you set the SetIdentityServerOrigin from reading Httpcontext.Request.Scheme which behind ASP.NET Core Module will always be http. Since all of your discovery endpoints etc are based on this value from GetIdentityServerBaseUrl they will also have a http schema. Please correct me if I am wrong or have missed a setting for this in IdentityServer 4.
Almost - IdentityServer relies on whatever is set for the host/protocol etc - at this point in the pipeline.
Behind a load balancer you typically have some middleware in the pipeline before IdentityServer that re-writes those values according to your deployment.