What mTLS certificates are required when the client is also the server?

I am looking to connect a server and a number of clients using mTLS in golang. On my server I would like to be able to generate certificates to put on all the clients so clients can talk to the server but clients cannot talk to each other. My clients however, expose the golang http server on port 80 and my server communicates with the clients through API requests rather than the other way around (technically you could question whether what I am calling a server is in fact a client, but it’s a single source for certificate generation and serves content to the clients so will stick with this naming for simplicity). How could this be set up? Which certificates would need to be generated and is it possible to have the clients listen for requests using certificates generated from a single server?

I will most likely be using SmallStep for certificate generation and SmallStep examples would be useful, but the general approach would be fine too and can then look at how to replicate it with SmallStep separately.

I have looked at some existing golang examples but they tend to be steered at different setups:

https://smallstep.com/hello-mtls/doc/server/go

https://venilnoronha.io/a-step-by-step-guide-to-mtls-in-go

Here is the code at the moment which I am quite sure is using the certificates incorrectly :

Client:

step ca certificate "localhost" client.crt client.key

In the below, cert and key variables are client.crt and client.key respectively.

caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM([]byte(cert))

// Read the key pair to create certificate
keyPair, err := tls.X509KeyPair([]byte(cert), []byte(key))
if err != nil {
log.Fatal(err)
}

transport = &http.Transport{
IdleConnTimeout:     transportIdleConnTimeout,
MaxIdleConnsPerHost: transportMaxIdleConnsPerHost,
TLSClientConfig: &tls.Config{
    RootCAs:      caCertPool,
    Certificates: []tls.Certificate{keyPair},
},

Server:

step ca certificate "pm1" server.crt server.key

In the below, cert and key variables are server.crt and server.key respectively.

caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM([]byte(cert))

cert, err := tls.X509KeyPair([]byte(cert), []byte(key))
if err != nil {
    log.Fatal("server: loadkeys: ", err)
}

tlsConfig := tls.Config{
    ClientCAs:    caCertPool,
    ClientAuth:   tls.RequireAndVerifyClientCert,
    Certificates: []tls.Certificate{cert},
    MinVersion:   tls.VersionTLS13,
}

I am not using ca.crt (step ca root ca.crt) at all right now.

Solution

What mTLS certificates are required when the client is also the server?

From server perspective:

each service requires it's own certificate, to serve HTTPS
a list of trusted certificates, used to authenticate the incoming requests (the list should include all the certificates used by all the authorised clients).

From client perspective:

the list of certificates for all the servers that you intend to communicate with over mTLS; clients won't trust unknown certificates(similar to a web browser); usually the CA that signed certificates for all services is used here, but you may also add them individually if really want to
its own certificate which is used to sign the outgoing requests.

clients using mTLS [...] expose the golang http server on port 80

The port( :80 or :443 etc.) and the protocols(http, https, gRPC etc) are different things. However, there is a strong convention to use http on port 80 and https on :443. If you don't care about conventions, nothing can stop you to use https on port :80. Because you want to achieve mTLS, you will have to listen on HTTPS on the servers.

[...]but it’s a single source for certificate generation and serves content to the clients so will stick with this naming for simplicity[...]and is it possible to have the clients listen for requests using certificates generated from a single server?

I am not sure I understand the premise here, but certificate generation isn't usually in the responsibility of the servers or clients that are supposed to use them to establish mutual trust.

I will assume you have a CA that can sign certificates for each of your web-servers and for simplicity will name them:

µA: must communicate with µB and µC
µB: must communicate only with µA
µC: must communicate only with µA

HTTPS and mTLS are again different things: the first is the protocol and the second is an authentication method. I will try to address them individually:

HTTPS:

To start listening on HTTPS, each micro-service which is supposed to accept connections, should start a listener and should load its own .crt and .key(I will revisit this section when configuring mTLS, but for the moment let's solve the HTTPS):

func main() {
    http.HandleFunc("/hello", HelloServer)
    // Replace 443 with 80 if you really don't care about conventions
    err := http.ListenAndServeTLS(":443", "µ<A|B|C>.crt", "µ<A|B|C>.key", nil)
    if err != nil {
        log.Fatal("ListenAndServe: ", err)
    }
}

mTLS:

mTLS authentication is performed mutually between a client(micro-service sending the request) and a server(micro-service receiving the request). Each micro-service can be a client, a server or both.

From client perspective:

the certificate presented by the server must be verified and trusted
the client must load it's own certificate and sign its outgoing requests

From server perspective:

the incoming requests must be signed by a trusted list of client(s) certificates or CAs
the response must be signed with a the server certificate, which should be trusted by the client that originated the request.

Client A perspective:

    // Client A perspective
    client := &http.Client{
        Transport: &http.Transport{
            TLSClientConfig: &tls.Config{
                // provide the CA that signed certificates that are presented
                // by servers B and C
                RootCAs: caCertPool,
                // provide the certificate for microservice A, which is
                // initiating the request
                Certificates: []tls.Certificate{cert}, 
            },
        },
    }

Server B or C perspective:

tlsConfig := &tls.Config{
    // add only microservice A certificate in this pool; this will allow A 
    // to connect to B and C, but won't allow B to C and C to B 
    ClientCAs: caCertPool, 
    ClientAuth: tls.RequireAndVerifyClientCert,
}
tlsConfig.BuildNameToCertificate()

server := &http.Server{
    Addr:      ":443", // use :80 if you don't care about conventions
    TLSConfig: tlsConfig,
}

server.ListenAndServeTLS("server.crt", "server.key")